Problem when using metadata of type DateTime with other metadata

[santcorc]

Elasticsearch Version

7.6.2

Installed Plugins

No response

Java Version

16.0.1

OS Version

5.13.0-30-generic #33~20.04.1-Ubuntu

Problem Description

Working in an integration between the context broker Orion-LD, FIWARE CYGNUS and elasticsearch, when I am creating an entity which has two or more metadata of different type (for example, Text or Number) works fine, but when creating an entity with one metadata with type DateTime, if there are more metadata on the same attribute the insertion fails and the attribute is not stored on the database.

Steps to Reproduce

This entity is stored correctly

{
    "id": "oneEntity",
    "type": "AType",
    "description": {
        "type": "Property",
        "value": "Description",
        "metadata1": {
        "type": "Text",
        "value": "Text"
        }, 
        "metadata2": {
        "type": "Number",
        "value": 23
        }
    }

While this other entity creation fails

{
    "id": "twoEntity",
    "type": "AType",
    "description": {
        "type": "Property",
        "value": "Description",
        "metadata1": {
        "type": "Text",
        "value": "Text"
        }, 
        "metadata2": {
        "type": "DateTime",
        "value": "2022-02-17T23:00:00.000Z"
        }
    }

Logs (if relevant)

db-elasticsearch    | {"type": "server", "timestamp": "2022-03-04T10:30:31,716Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "[test-2022.03.04][0] failed to execute bulk item (index) index {[test-2022.03.04][cygnus_type][1646389829068-025CEBAC466474149DAE5682DB58A6AC], source[_na_]}", "cluster.uuid": "C6XE81jNTy2sRMQpHfyiSg", "node.id": "yCzki8g2QLurGURcmnEW0Q" , 
db-elasticsearch    | "stacktrace": ["java.lang.IllegalArgumentException: mapper [attrMetadata.value] of different type, current_type [date], merged_type [text]",
db-elasticsearch    | "at org.elasticsearch.index.mapper.FieldMapper.doMerge(FieldMapper.java:359) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch    | "at org.elasticsearch.index.mapper.DateFieldMapper.doMerge(DateFieldMapper.java:609) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch    | "at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:346) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch    | "at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:56) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch    | "at org.elasticsearch.index.mapper.DocumentParser.createDynamicUpdate(DocumentParser.java:237) ~[elasticsearch-7.6.2.jar:7.6.2]",

[nik9000]

I think this is just how Elasticsearch works. When Elasticsearch first sees a field it takes a guess about what type it should have. I'm not sure about what you are doing exactly, but it sure looks like this.

This really is more of a "how do I use Elasticsearch?" question than a bug. Probably obscured by some other software. I'm guessing you should take up your problem with them. If you have other questions about Elasticsearch though you should probably open them at https://discuss.elastic.co/

[jandres6579]

The cygnus component sends the following to elasticsearch:

fiware-cygnus | time=2022-03-16T16:00:29.874Z | lvl=INFO | corr=175d81b1-d93b-402b-9ffc-5171dc3cabf2 | trans=175d81b1-d93b-402b-9ffc-5171dc3cabf2 | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=run | msg=com.telefonica.iot.cygnus.sinks.NGSIElasticsearchSink$1[111] : [elasticsearch-sink] Persisting data at NGSIElasticsearchSink. (index=cygnus-fiware-service--urn-ngsi-ld-room-room3070-room-2022.03.16, type=cygnus_type, data=[{data={"recvTime":"2022-03-16T16:00:29.054+0000","entityType":"Room","attrMetadata":[],"entityId":"urn:ngsi-ld:Room:Room3070","attrValue":"21","attrName":"temperature","attrType":"Number"}, recvTimeTs=1647446429054}, {data={"recvTime":"2022-03-16T16:00:29.054+0000","entityType":"Room","attrMetadata":[{"name":"amountOfThisGood","type":"Text","value":"1"},{"name":"timestamp","type":"DateTime","value":"2022-03-15T23:00:00.000Z"}],"entityId":"urn:ngsi-ld:Room:Room3070","attrValue":"803","attrName":"pressure","attrType":"Number"}, recvTimeTs=1647446429054}])

And in ElasticSearch component I obtain:

db-elasticsearch | {"type": "server", "timestamp": "2022-03-16T16:00:29,784Z", "level": "DEBUG", "component": "o.e.c.c.C.CoordinatorPublication", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "publication ended successfully: Publication{term=1, version=22}", "cluster.uuid": "LfRZ0XnjRpe5L3H1XZYQpQ", "node.id": "F4TG9aAHRqCMAWafV3DtDw" }
db-elasticsearch | {"type": "server", "timestamp": "2022-03-16T16:00:29,785Z", "level": "DEBUG", "component": "o.e.c.s.MasterService", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "took [0s] to notify listeners on successful publication of cluster state (version: 22, uuid: 2vVUuoMXRN-JveMP1-44Uw) for [put-mapping[cygnus_type]]", "cluster.uuid": "LfRZ0XnjRpe5L3H1XZYQpQ", "node.id": "F4TG9aAHRqCMAWafV3DtDw" }
db-elasticsearch | {"type": "server", "timestamp": "2022-03-16T16:00:29,820Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "docker-cluster", "node.name": "elasticsearch", "message": "[cygnus-fiware-service--urn-ngsi-ld-room-room3070-room-2022.03.16][0] failed to execute bulk item (index) index {[cygnus-fiware-service--urn-ngsi-ld-room-room3070-room-2022.03.16][cygnus_type][1647446429054-890AC15FB9F056E7C2B84D93216A9091], source[na]}", "cluster.uuid": "LfRZ0XnjRpe5L3H1XZYQpQ", "node.id": "F4TG9aAHRqCMAWafV3DtDw" ,
db-elasticsearch | "stacktrace": ["java.lang.IllegalArgumentException: mapper [attrMetadata.value] of different type, current_type [date], merged_type [text]",
db-elasticsearch | "at org.elasticsearch.index.mapper.FieldMapper.doMerge(FieldMapper.java:359) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.mapper.DateFieldMapper.doMerge(DateFieldMapper.java:609) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:346) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:56) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.mapper.DocumentParser.createDynamicUpdate(DocumentParser.java:237) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:83) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:267) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:793) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:770) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:742) ~[elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:267) [elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:157) [elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]",
db-elasticsearch | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
db-elasticsearch | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
db-elasticsearch | "at java.lang.Thread.run(Thread.java:830) [?:?]"] }