Indexing a composite record

[ronag]

This is related to #211.

Consider a model A: { B, C } when B and C are references to other records i.e.

A: {
  B: 'a/1',
  C: 'b/2'
}
B: {
  value: 'foo'
}
C: {
  value: 'bar'
}

Now I want to index this into e.g. elastic search. This means (if not using elastic search parent-child) that I have to build the whole object:

A: {
  B: { value: 'foo' },
  C: { value: 'bar' }
}

In order to do this I have to listen on the patterns a|b|c/* and whenever one of them changes I have to request each child or parent, build the object, and index it into elastic search.

Finding the parent for B and C is the first problem, but can be resolved with some id hackery or using a 'parent' field.

This would work fine if I could guarantee that a/*,b/*,c/* all happen on the same client/listener. However, that kills the purpose of #211. However, with #211 the different parts of an object could be listened on different shards which will lead to a lot of contention.

I am unsure how to resolve this. I don't have any good ideas either. Elastic search solves this problem internally by using parent-child mapping which ensure that all parent/child are invoked on the same shard/listener/indexer. So we will probably get around the issue using elastic search mappings.

[ronag]

This goes a bit into the #338 story of indexing from deepstream into other services, and not the other way around.