Need information model for directory

[mmccool]

We need an overall information model for all the data in a directory. Does it include information besides TDs? Eg out-of-band information (TTL, update timestamps, location...), links (eg to other directories), Thing Models, etc.
Can SPARQL queries be over the entire model?

Related issues:

• Federation: How do we handle federation in SPARQL queries #97
• Out-of-band info: Define Out-of-Band Information #30
• Links: Add links section to directory information model #34
• TTL: Consider TTL Use cases #18
• Geolocation: Geolocation
• Timestamps: Timestamps

[mmccool]

To be clearer, I'm thinking each TD record would "internally" look like this:

<td_record> :=
{
    // metadata
   "ttl": <ttl>,
   "location": <location>,
   "lastUpdated": <timestamp>,
   ...
    // tds (not modified by directory)
    "td": <parsed TD JSON>,
    "original": <raw string, for signing preservation purposes>
}

and the directory would basically stored as a JSON-LD object containing a set of these:

<directory> := { 
  "@context": <directory ontology>,
   <local_id>: <td_record>, ...
}

See however discussion in #18, which already discussed something similar to this (and the issues with it...). In particular we might want to link to tds rather than embedding them, and need to look into JSON-LD friendliness. @farshidtz will look at the above issue and try to extract the various options and their pros and cons.

[farshidtz]

I provided a summary of all proposals in my presentation (slides 9-11) back in June 2020 F2F. But an up-to-date summary would be more useful.

For the API spec, it is important to know how the directory exposes the resources, and that is what I will cover below. I will create separate posts for the three proposals to allow 👍 👎 votes. Exposing the collection of resources and pagination is another issue: #16

I will only include created and modified as out-of-band attributes but there could be others. I will particularly leave out ttl and expires because as suggested by @benfrancis, we could rely on protocol-specific mechanisms to convey these from client to server and vice versa for individual resources. In such case, it is safe to assume that all out-of-band information passed in response bodies are system-generated and read-only.

[farshidtz]

1. Wrapped TD

GET /td-wrapper/urn:example:1234

{
  "@context":"https://www.w3.org/2019/wot/tdd/v1",
  "id":"urn:example:1234", // this may or may not be equal to td.id
  "created": "2020-06-10T09:00:00Z",
  "modified":"2020-06-12T10:00:01Z",
  "td": {
    "@context":"https://www.w3.org/2019/wot/td/v1",
    "id":"urn:example:1234",
    "title":"ExampleSensor",
    "created":"2020-05-01T08:00:00Z",
    "modified":"2020-06-12T10:00:00Z",
    ...
  }
}

• This is a new resource which wraps the TD. Let's call this "TD Wrapper" due to the lack of better naming.
• The resource location (HTTP path) includes the TD Wrapper's ID.
• Media type is not application/td+json.
• If TD already has a unique ID, the TD Wrapper's ID will be redundant.
• Consumers submit and retrieve the TD Wrapper.
• The TD object can be signed and verified independent of the wrapper.

[farshidtz]

2. Enriched TD

GET /td/urn:example:1234?registration_info=true

{
  "@context":[
    "https://www.w3.org/2019/wot/td/v1",
    {"tdd": "https://www.w3.org/2019/wot/tdd/v1"},
  ],
  "id":"urn:example:1234",
  "title":"ExampleSensor",
  "created":"2020-05-01T08:00:00Z",
  "modified":"2020-06-12T10:00:00Z",
  ...
  "tdd:info":{ // present because of the query parameter (registration_info=true)
    "id":"urn:example:1234", // this may or may not be equal to td.id
    "created": "2020-06-10T09:00:00Z",
    "modified":"2020-06-12T10:00:01Z"
  }
}

• This is a TD with extensions inserted inside upon request. In this example, the extensions are all inside an object, but we can have them at the top level too (e.g. tdd:created, tdd:modified); we can discuss further if this proposal is selected.
• The resource location (HTTP path) includes the TD's ID and when not present, the system-generated UUIDv4 URN given to the consumer upon registration.
• Media type is application/td+json.
• Consumers submit and retrieve the TD. They may retrieve the Enriched TD.
• The context can be simplified if the hypothetical tdd context gets merged into TD's.
• If signed, the verification should be done on the non-Enriched TD, either by removing extensions or querying just the TD (See Canonicalization requirements for directories #104 for TD canonicalization)

[farshidtz]

3. Decoupled Resources

GET /td/urn:example:1234

{
  "@context":"https://www.w3.org/2019/wot/td/v1",
  "id":"urn:example:1234",
  "title":"ExampleSensor",
  "created":"2020-05-01T08:00:00Z",
  "modified":"2020-06-12T10:00:00Z",
  ...
}

• Media type is application/td+json

GET /td-info/urn:example:1234

{
  "@context":"https://www.w3.org/2019/wot/tdd/v1",
  "id":"urn:example:1234", // this may or may not be equal to td.id
  "created": "2020-06-10T09:00:00Z",
  "modified":"2020-06-12T10:00:01Z",
  "link":"/td/urn:example:1234” // relative link to TD
}

• This is a new resource containing all the out-of-band information. I'll call this resource "TD Info" for now. @mjkoster suggested "Index", but I this that was for the whole list in the <directory> example in the previous comment.
• This resource could be represented as a TD, but we can discuss it if this proposal is selected.
• If this resource (or parts of it) is created by a consumer, rather than automatically generated along with a TD, then the consumer must submit it separately.
• No easy way to query whole TDs while filtering based on this information.

[AndreaCimminoArriaga]

I think having an information model to express data regarding directories is a good idea. If the model is expressed as an ontology, SPARQL queries could refer to those terms. Additionally, those terms could be also useful for the federation, e.g., if we know that a known TDD has a 'response ratio' very low we could avoid including such TDD in the federation.

The best way to proceed in case of expressing these data as an ontology, would be to develop a TDD ontology that imports the TD one and extends it.

[mmccool]

MAYBE we can use TD Links for metadata... to discuss.

[mmccool]

So I'm personally leaning towards option 2 at this point (because it satisfies both "being RESTful" and "supports SPARQL queries on the metadata") but think we should give it another week to collect votes. Let's plan to make a decision in the Jan 11 meeting.

[ashimura]

During the call on 11 Jan 2021, we agreed option 2 would be the best, and would like to think about detailed data flow based on some concrete use case(s).

Note that this issue is related to issue #54 on possible additional vocabulary.

[mmccool]

Changed title to "refine" since now we have one, but it may need some improvements.
Note: this is about the "internal" information model of the directory, see also #116 as there may be other information models needed for "interaction views".

[AndreaCimminoArriaga]

I did a proposal for the pagination in issue #54, this solution can resolve the problem of nested contexts, and also, the possible namespace collision that may occur using TDs extended with other ontologies.

[farshidtz]

From the Discovery call:

• The information model is already described in the specification.