Implement a timestamp [pending: attribute creation/modification dates]

[gogos-venge]

Implement an automatic timestamp (edit: of last action, or let a user set at update/append). That would save lives considering syncing.

{
  "contextElements": [
  {
    "type": "Room",
    "isPattern": "false",
    "id": "Room1",
    "timestamp": "1428799422",
    "attributes": [
    {
      "name": "temperature",
      "type": "float",
      "value": "26.5",
      "timestamp": "1428799841",
      "metadatas": [
      {
        "name": "average",
        "type": "float",
        "value": "22.4",
        "timestamp": "1428799831"
      },
      {
        "name": "accuracy",
        "type": "float",
        "value": "0.9",
        "timestamp": "1428799312"
      }
      ]
    }
    ]
  }
  ],
  "statusCode": {
    "code": "200",
    "reasonPhrase": "OK"
  }
}

[fgalan]

Thanks for the feedback!

Actually, this is a planned feature since time ago, although implementing it in a slightly different way. Instead of including a timestamp field (which is not allowed by NGSI spec) to specify the timestamp of an attribute we plan to use a metadata. We don't plan to implement a timestamp for the metadata themselves, but we think that doing it at attribute level is enough (in some sense, metadata array is part of the value of the attribute).

[gogos-venge]

Yes, I thought later after posting this feedback, NGSI would not allow such thing. But actually I already implemented exactly what you said for an API I'm working, which uses Orion for syncing. I figured out it would be impossible to set a timestamp for metadata since it's the bottom level, and using metadata pairs would not only be a VERY bad practice, but would eventually generate extra traffic and frustration. That disables metadatas for us though, but still attributes should be quite enough for the data storing.

[fgalan]

Finally, we have defered this to v2 API. This version uses a simplified API such as:

  {
    "type": "Car",
    "id": "P-9873-K",
    "speed": {
      "value": 100,
      "accuracy": 2
      }
  }

In which including creDate and modDate (both at entity level and attribute level) would be easy:

  {
    "type": "Car",
    "id": "P-9873-K",
    "creDate": <ISODATE>,
    "modeDate": <ISODATE>,
    "speed": {
      "value": 100,
      "creDate": <ISODATE>,
      "modeDate": <ISODATE>,
      "accuracy": 2
      }
  }

By default creDate/modDate are not returned (thus the JSON is simpler, in a general case). If the client wants to get the information, then some URI parameter has to be used. This is currently under discussion, but a possibility could be:

?creDate=true&modDate=true

[gogos-venge]

creDate and modeDate. That creates a new whole level of usability. Excellent work once again.

[fgalan]

Well... the issue has not been actually solved, only a solution proposed :) Thus, I'm reopening it.

[fgalan]

Entity creation and modification times implemented in PR #1829 (look the documentation pieces in that PR in order to see how it works at the end).

Not sure yet what to do with attribute modification and creation times. By the moment, this issue remains opened while that piece of work/decission is pending.

[jmcanterafonseca]

some comments:

I think we need to adopt a naming schema for those properties which are automagically added by the implementation but which do not actually are part of the entity or the attribute themselves. That will help developers to distinguish them. One option is to use something "a la python" such as

__dateCreated__
__dateModified__

and avoiding any kind of clash between user-defined data and these attributes. Another option, I'm more in favour, is to use '@dateCreated' '@dateModified' as names for these attributes which will be more aligned with our upcoming adoption of JSON-LD.

In any case I think the name of the properties should be 'dateCreated' and 'dateModified' (to align to schema.org).

With regards to attributes I think these could be part of the metadata associated to the attribute, but right now metadata layer is under discussion. So question mark here.

With regards to the API to ask for their rendering I would suggest, for the sake of orthogonality, not to have any particular option but the regular patterns to do it. With the note that @dateCreated and @dateModified would not be rendered by default.

/v2/entities?id=1234&attrs=@dateCreated,@dateModified
/v2/entities?id=1234&attrs=myAttr.@dateCreated

I think right now we do not have the possibility to select partial content from a compound attribute but that is something could be considered as well.

[fgalan]

What about ngsiv2:dateCreated and ngsiv2:dateModified (aligned with geo:point and other geo-stuff)? Comparing:

GET /v2/entities?id=1234&orderBy=@dateCreated,!@dateModified

and

GET /v2/entities?id=1234&orderBy=__dateCreated__,!__dateModified__

with

GET /v2/entities?id=1234&orderBy=ngsiv2:dateCreated,!ngsiv2:dateModified

I think the third option has less "syntactical awkwardness".

Regarding orthogonality, we can use an alternative to options to specify these attributes has to be rendered in the response. However, attrs cannot be used for that, as if we use:

GET /v2/entities?id=1234&attrs=dateCreated,dateModified

you will only get dateCreated and dateModified and no other attribute, as attrs acts as a filter. Thus, we need something not only orthogonal to options but also orthogonal to attrs. The natural choice would be a new URL param for these (an maybe others) virtual attributes, e.g. extraAttrs. This way, we can have things like this:

GET /v2/entities?id=1234&extrAttrs=dateCreated&attrs=temperature,dateCreated

meaning that we wan to include dateCreated in the response and that we want filter all except 2 attributes: temperature and dateCreated.

Finally, a side-note about metadata. Metadata is under discussion only in the conext of the JSON-LD encoding discussion. I mean, the support and usage of metadata in the "regular" NGSIv2 API encodings is not under discussion. Thus, your comments about using metadata for rendering creation/modification dates about attributes do fully apply (this has been deferred to a post-0.28.0 version).

[fgalan]

Most of the functionality has been implemented in PR #2590. However, the support for filtering in date metadata has not been yet implemented/checked.

The following .test should be used to implement that part. No aditional .test appart from them should be needed.

• cases/0876_attribute_dates/filter_attr_datecreated_sub.test.DISABLED
• cases/0876_attribute_dates/filter_attr_dates.test.DISABLED
• cases/0876_attribute_dates/filter_attr_datemodified_sub.test.DISABLED

Currently they are emptu files. In order to complete them the same script that the one used in the following .tests should be used, but applying it to attribute dates as metadata (instead of entity dates as attributes):

• cases/0876_entity_dates/filter_datecreated_sub.test
• cases/0876_entity_dates/filter_entity_dates.test
• cases/0876_entity_dates/filter_datemodified_sub.test

[fgalan]

The support for filtering in date metadata implemented in PR #2615

[fgalan]

Ready for QA testing.

[iariasleon]

Covered E2E in features/pepflows/010_pep2cbv2_virtualattributes.feature

[fgalan]

Issues in milestone QATestPendingE2ECovered are going to be closed (it is not realistic to think we would have people working on behave testing at the present moment). However, issue is labeled with QATestPending label in the case things change and we need to recover it.