Add Presence functionality

[alecgibson]

This change adds the ability for clients to broadcast information about
"Presence" - the notion of a client's position or state in a particular
document. This might be represent a cursor in a text document, or a
highlighted field in a more complex JSON document, or any other
transient, current information about a client that shouldn't necessarily
be stored in the document's chain of ops.

The main complication that this feature solves is the issue of keeping
presence correctly associated with the version of a Doc it was created
at. For example, in a "naive" implementation of presence, presence
information can arrive ahead of or behind ops, which - in a text-based
example - can cause the cursor to "jitter" around the change. Using the
ShareDB implementation will ensure that the presence is correctly
transformed against any ops, and will ensure that presence information
is always consistent with the version of the document. We also locally
transform existing presence, which should help to keep (static) remote
presence correctly positioned, independent of latency.

In order to facilitate this, the feature must be used with an OT type
that supports presence. The only requirement for supporting presence is
the support of a transformPresence method:

type.transformPresence(presence, op, isOwnOperation): presence;

• presence Object: the presence data being transformed. The type
  will define this shape to be whatever is appropriate for the type.
• op Op: the operation against which to transform the presence
• isOwnOperation: boolean: whether the presence and the op have the
  same "owner". This information can be useful for some types to break
  ties when transforming a presence, for example as used in
  rich-text

This work is based on the work by @gkubisa and @curran, but with
the following aims:

• avoid modifying the existing Doc class as much as possible, and
  instead use lifecycle hooks
• keep presence separate as its own conceptual entity
• break the presence subscriptions apart from Doc subscriptions
  (although in practice, the two are obviously tightly coupled)
• allow multiple presences on a single Doc on the same Connection

[coveralls]

Coverage increased (+0.5%) to 97.34% when pulling 929d515 on yet-another-presence into eee5acc on master.

[curran]

Woohoo! Thrilled to see this fresh take on presence.

[alecgibson]

Let's got a step further here, and completely remove the notion of presence from Doc (ie Doc has no presence-related methods or properties). Instead, let's have an intermediary notion of DocPresence, which exists separately to Doc, but is responsible for handling all state of tracking remote presence, and also of creating instances of local presence. In this way, we can easily handle our presence subscription without needing any instances of local presence, but also have none or multiple local presences.

We can also completely decouple presence from Docs altogether, by having a notion of subscribing to presence channels (eg mouse cursor position on a screen might be a notion of presence that is not coupled to a Doc). The Doc presence should be an extension of this more generic presence.

So the API might look something like:

const presence = connection.getDocPresence(collection, id);
// Start listening for presence
presence.subscribe((err) => {...});
// Handle incoming presence (eg update a cursor position)
presence.on('receive', (remotePresence) => {...});
// Create a instances of local presence
const localPresence1 = presence.create(id1);
const localPresence2 = presence.create(id2);
// Presences can send independently
localPresence1.submit({...});
localPresence1.destroy((err) => {...});
// Can unsubscribe, but still broadcast
presence.unsubscribe((err) => {...});
localPresence2.submit({...});
// Broadcast a removal of all local presence, and free up local memory
presence.destroy((err) => {...});

[alecgibson]

I think the one thing I still find a little bit weird about its current incarnation is that if you're not subscribed, you won't receive any requests for your presence, so you won't inform any new subscribers. For example:

• client 1 is subscribed
• client 2 is not subscribed, and submits presence
• client 1 receives client 2's presence
• client 3 connects and subscribes, requesting all current presence
• client 2 is not subscribed, so doesn't receive the request, and doesn't tell client 3 its old presence (despite having submitted it)

Maybe this is an edge case we don't need to think about too hard? The mainline case will presumably be all clients being subscribed all the time.

I guess the other thing to think about is potentially garbage collecting stale clients who disconnected abruptly without a chance to update other clients. Maybe we can handle this in the Agent on the stream close? If the Agent stored all local presences associated with a connection, it could then broadcast an update to remote clients saying that the client has disconnected. NB: We'd potentially want debouncing if a client is on a flaky connection? So we could just push all of this onto the consumer to use a TTL? (Also handles the case where a user just leaves their browser open for days on end without actually being in the tab)

[curran]

Greetings @alecgibson

I'm looking at this work with fresh eyes and would like to try it out. Before diving into trying to craft a working example myself I thought I'd ask, do you have any working examples that use this implementation for presence features?

Thanks!

[curran]

Also @nateps @ericyhwang I'm curious if there is a possibility that this implementation might get merged at some point. I know you've got a lot to juggle but just curious on a high level, is the possibility there? I'm again evaluating the @teamwork/sharedb vs sharedb packages and am tentatively switching back to mainline sharedb to get some of the recent fixes published. Having presence implemented here would really seal the deal!

[alecgibson]

@curran I haven't got an example working at the moment, because the API was a bit unstable through a couple of PR meetings. However, if you have a look at the tests, the usage should hopefully be pretty self-evident. Note that the README in this PR hasn't be correctly updated (this is also a note to self to update!). I haven't made the last couple of meetings, and with the holidays coming up, probably won't get much more done on this until the new year, but I am trying to push ahead with this! This is 100% something we need to use, so I'll be seeing it through.

If you want to try to get something up and running, I'd love the feedback. A very quick overview of the API as it currently stands (NB: using async / await syntax out of sheer laziness for not nesting callbacks on Github):

const doc = connection.get('books', 'northern-lights');
await doc.subscribe();

// This presence object represents all presence on the Doc. Can have multiple remote and
// local presences, available on `remotePresences` and `localPresences` properties
const presence = connection.getDocPresence('books', 'northern-lights');
presence.on('receive', (id, remotePresence) => {/** handle presence **/});
await presence.subscribe();

const localPresence = presence.create('user-1');
localPresence.submit({...});

The main piece of the puzzle that's missing is a type that supports presence. All it needs to do is support transformPresence. If you want to play around with rich-text, I've got a fork that should work with this presence PR here (note that you need to be on the presence branch). Usage with this type would look something like:

localPresence.submit({index: 4, length: 0}); // uses the same structure as Quill
presence.on('receive', (id, remotePresence) => {
  // Handle presence.
  // remotePresence will have the same structure as above: {index: number; length: number}
});

Since the transformPresence method name is the same as #288 you may also be able to use types that were compatible with that.

[alecgibson]

Let's think about if we can get rid of this extra src emission. We basically need to know if the op is our own op in RemoteDocPresence._handleOp, and in Backend.transformPresenceToLatestVersion

[alecgibson]

Let's keep this, but bury it one level deep in an object with a src property for forwards-compatibility and self-documentation. We should also update the README.

[alecgibson]

@ericyhwang this should be ready for review. I've updated the README, and added an example to the examples folder if you want to play around with it (cc @curran ).

I think the only outstanding issue is figuring out what to do with emitting src on op (which I'm assuming we'll discuss in the meeting next week).

[curran]

Woohoo! Looking forward to giving the examples a spin. Will let you know how it goes.

[ericyhwang]

LGTM! Let's finally get this merged in 🚀

Nate says he's OK with adding you as an owner to the NPM sharedb package, so I've gone and done so.

It's nearing the end of the workday for me, so I'm not going to merge+publish now. If you'd like to do the merge and publish in the morning your time, go for it!

Otherwise, we can do so during the meeting tomorrow and write release notes. I'd like to be sure and credit Greg and Curran for their work on presence, too.

[gkubisa]

Great work guys! I'm really happy to see this feature complete. 🎉

[curran]

Amazing! Really really excited about this. Thank you everyone for your hard work here.

[neilcarv]

Hey guys, this work is brilliant. A quick question, I can see a PR for json0 and an issue open for json1 to support a transformPresence method. Would that be something that will be merged/added soon? Also, in general, should json0 or json1 be used?

Thanks, Neil

[curran]

Hi @neilcarv ! This is something near and dear to my heart and I'm glad you asked.

Current status is: there's no demo code for the new implementation of ShareDB presence used with JSON0. I plan to work on one soon, but haven't been able to dig into it yet.

Here's a PR agains JSON0 that implements the OLD version of presence https://github.com/ottypes/json0/pull/31/files . I would love to see this updated to the presently published API. I'm not sure how far it will diverge, but hopefully getting it to work should be relatively painless.

I'd also like to see this demo ported to use the new presence implementation https://github.com/datavis-tech/json0-presence-demo

I hope to take a stab at this some time but if anyone else has interest, feel free to have a crack at it!

Re: JSON0 vs. JSON1 - folks generally are still using JSON0. I don't know of anyone actively using JSON1. It looks really nice, but there's more of an ecosystem around JSON0.

/cc @houshuang