Clarify use of sync. Closes #2298

docs/Information-Flow.md

@@ -70,25 +70,115 @@ In our example above, the name of the *contract action* generated by `sbp('gi.ac
 
 ##### Subscribing to a contract
 
-Subscribing to a contract and syncing all of its updates is done by calling `'chelonia/contract/sync'`:
+Chelonia implements _reference counting_ to automatically manage contract
+subscriptions. When the reference count is positive, a contract subscription
+is created, meaning that the contract will be synced and Chelonia will listen
+for new updates. When the reference count drops to zero, the contract will be
+automatically removed(*).
+
+Subscribing to a contract and syncing all of its updates is done by calling `'chelonia/contract/retain'`:
 
 For example:
 
 ```js
-sbp('chelonia/contract/sync', contractID)
+sbp('chelonia/contract/retain', contractID)
 
 // OR wait until it finishes syncing:
 
-await sbp('chelonia/contract/sync', contractID)
+await sbp('chelonia/contract/retain', contractID)
 ```
 
 This will subscribe us to the contract and begin listening for new updates.
 
+When the contract is no longer needed, you can call `'chelonia/contract/release'`.
+
+For example:
+
+```js
+await sbp('chelonia/contract/release', contractID)
+```
+
+###### When to call `retain` and `release`
+
+Normally, you should call `retain` each time an event is about to happen that
+requires receiving updates for a contract. You should then call `release` when
+that reason for subscribing no longer holds.
+
+**IMPORTANT:** Each call to `release` must have a corresponding call to `retain`.
+In other words, the reference count cannot be negative.
+
+Three examples of this are:
+
+  * **Writing to a contract.** Writing to a contract requires being subscribed
+    to it, so you should call `retain` before sending an action to it. The
+    contract can be released by calling `release` after the writes have completed.
+  * **Subscribing to related contracts in side-effects.** A common use case for
+    calling `retain` and `release` is when a contract is related to other
+    contracts. For example, you could have users that can be members of different
+    groups. Every time a user joins a group, you would call `retain` in the
+    side-effect of that action, and then call `release` when the group membership
+    ends.
+  * **Logging a user in.** If your application has users that are represented by
+    contracts, those contracts usually need to be subscribed during the life of
+    a session. This would be an example where you would have a call to `retain`
+    when the account is created and then there could be no `release` call.
+
+###### Ephemeral reference counts
+
+Chelonia maintains two different reference counts that you can directly control
+using `retain` and `release`: ephemeral and non-ephemeral.
+
+Non-ephemeral reference counts are stored in the root Chelonia state and are
+meant to be persisted. In the examples above, the examples of subscribing to
+related contracts and logging a user in would fall in this category. If Chelonia
+is restarted, you want those references to have the same value.
+
+On the other hand, _ephemeral_ reference counts work a differently. Those are
+only stored in RAM and are meant to be used when restarting Chelonia should
+_not_ restore those counts. The example of writing to a contract would be one
+of those cases. If Chelonia is restarted (for example, because a user refreshes
+the page) and you're in the middle of writing to a contract, you would not want
+that reference to persist because you'd have no way of knowing you have to call
+`release` afterwards, which would have the effect of that contract never being
+removed.
+
+All calls to `retain` and `release` use non-ephemeral references by default. To
+use ephemeral references, pass an object with `{ ephemeral: true }` as the last
+argument. Note that ephemeral `retain`s are paired with ephemeral `release`s,
+and non-ephemeral `retain`s are paired with non-ephemeral `release`s (i.e.,
+don't mix them).
+
+For example,
+
+```js
+await sbp('chelonia/contract/retain', contractID, { ephemeral: true })
+try {
+  // do something
+} finally {
+  await sbp('chelonia/contract/release', contractID, { ephemeral: true })
+}
+```
+
+###### `chelonia/contract/sync`
+
+In addition to `retain` and `release`, there is another selector that's
+relevant: `chelonia/contract/sync`. You use `sync` to force fetch the latest
+state from the server, or to create a subscription if there is none (this is
+useful when bootstrapping your app: you already have a state, but Chelonia
+doesn't know you should be subscribed to a contract). `sync` doesn't affect
+reference counts and you should always call `sync` on contracts that have at
+least one refeence. This means that you need to, at some point, have called
+`retain` on that contract first.
+
 (...WIP...)
 
 When subscribed to a Contract, the user is updated each time an action there is called, even if the action wasn't triggered by the user itself. (TODO: Add link/reference to where this happens)
 
 So you don't need to worry about this for now, it just works 🔮.
 
+(*) The actual mechanism is more involved than this, as there are some other
+reasons to listen for contract updates. For example, if contracts use foreign
+keys (meaning keys that are defined in other contracts), Chelonia may listen for
+events in those other contracts to keep keys in sync.
 
-That's all for now! Feel free to dive even more deeply in the files mentioned so far and complement these docs with your discoveries.
+That's all for now! Feel free to dive even more deeply in the files mentioned so far and complement these docs with your discoveries.

frontend/controller/actions/identity.js

@@ -170,35 +170,39 @@ export default (sbp('sbp/selectors/register', {
         hooks: {
           postpublishContract: async (message) => {
             // We need to get the contract state
-            await sbp('chelonia/contract/sync', message.contractID())
-
-            // Register password salt
-            const res = await fetch(`${sbp('okTurtles.data/get', 'API_URL')}/zkpp/register/${encodeURIComponent(username)}`, {
-              method: 'POST',
-              headers: {
-                'authorization': await sbp('chelonia/shelterAuthorizationHeader', message.contractID()),
-                'content-type': 'application/x-www-form-urlencoded'
-              },
-              body: new URLSearchParams({
-                'r': r,
-                's': s,
-                'sig': sig,
-                'Eh': Eh
+            await sbp('chelonia/contract/retain', message.contractID(), { ephemeral: true })
+
+            try {
+              // Register password salt
+              const res = await fetch(`${sbp('okTurtles.data/get', 'API_URL')}/zkpp/register/${encodeURIComponent(username)}`, {
+                method: 'POST',
+                headers: {
+                  'authorization': await sbp('chelonia/shelterAuthorizationHeader', message.contractID()),
+                  'content-type': 'application/x-www-form-urlencoded'
+                },
+                body: new URLSearchParams({
+                  'r': r,
+                  's': s,
+                  'sig': sig,
+                  'Eh': Eh
+                })
               })
-            })
 
-            if (!res.ok) {
-              throw new Error('Unable to register hash')
-            }
+              if (!res.ok) {
+                throw new Error('Unable to register hash')
+              }
 
-            userID = message.contractID()
-            if (picture) {
-              try {
-                finalPicture = await imageUpload(picture, { billableContractID: userID })
-              } catch (e) {
-                console.error('actions/identity.js picture upload error:', e)
-                throw new GIErrorUIRuntimeError(L('Failed to upload the profile picture. {codeError}', { codeError: e.message }), { cause: e })
+              userID = message.contractID()
+              if (picture) {
+                try {
+                  finalPicture = await imageUpload(picture, { billableContractID: userID })
+                } catch (e) {
+                  console.error('actions/identity.js picture upload error:', e)
+                  throw new GIErrorUIRuntimeError(L('Failed to upload the profile picture. {codeError}', { codeError: e.message }), { cause: e })
+                }
               }
+            } finally {
+              await sbp('chelonia/contract/release', message.contractID(), { ephemeral: true })
             }
           }
         },
@@ -243,10 +247,10 @@ export default (sbp('sbp/selectors/register', {
       } else {
         // If there is a state, we've already retained the identity contract
         // but might need to fetch the latest events
-        await sbp('chelonia/contract/sync', identityContractID, { force: true })
+        await sbp('chelonia/contract/sync', identityContractID)
       }
     } catch (e) {
-      console.error('Error during login contract sync', e)
+      console.error('[gi.actions/identity] Error during login contract sync', e)
       throw new GIErrorUIRuntimeError(L('Error during login contract sync'), { cause: e })
     }
 
@@ -346,7 +350,6 @@ export default (sbp('sbp/selectors/register', {
       //      queues), including their side-effects (the `${contractID}` queues)
       //   4. (In reset handler) Outgoing actions from side-effects (again, in
       //      the `encrypted-action` queue)
-      cheloniaState = await sbp('chelonia/rootState')
       await sbp('okTurtles.eventQueue/queueEvent', 'encrypted-action', () => {})
       // reset will wait until we have processed any remaining actions
       cheloniaState = await sbp('chelonia/reset', async () => {

frontend/controller/actions/index.js

@@ -208,10 +208,5 @@ sbp('sbp/selectors/register', {
         }
       })
     }
-
-    // TODO: Temporary sync until the server does signature validation
-    // This prevents us from sending messages signed with just-revoked keys
-    // Once the server enforces signatures, this can be removed
-    await sbp('chelonia/contract/sync', contractID, { force: true })
   }
 })

frontend/controller/actions/utils.js

@@ -310,7 +310,8 @@ export function groupContractsByType (contracts: Object): Object {
   const contractIDs = Object.create(null)
   if (contracts) {
     // $FlowFixMe[incompatible-use]
-    Object.entries(contracts).forEach(([id, { type }]) => {
+    Object.entries(contracts).forEach(([id, { references, type }]) => {
+      if (!references) return
       if (!contractIDs[type]) {
         contractIDs[type] = []
       }
@@ -343,7 +344,7 @@ export async function syncContractsInOrder (groupedContractIDs: Object): Promise
       // Sync contracts in order based on type
       return getContractSyncPriority(a) - getContractSyncPriority(b)
     }).map(([, ids]) => {
-      return sbp('chelonia/contract/sync', ids, { force: true })
+      return sbp('chelonia/contract/sync', ids)
     }))
   } catch (err) {
     console.error('Error during contract sync (syncing all contractIDs)', err)

frontend/controller/app/identity.js

@@ -324,7 +324,7 @@ export default (sbp('sbp/selectors/register', {
         sbp('okTurtles.events/off', LOGIN_ERROR, loginErrorHandler)
 
         const errMessage = e?.message || String(e)
-        console.error('Error during login contract sync', e)
+        console.error('[gi.app/identity] Error during login contract sync', e)
 
         const promptOptions = {
           heading: L('Login error'),

frontend/main.js

@@ -188,7 +188,7 @@ async function startApp () {
           // since they refer to Vuex (i.e., tab / window) state and not to
           // Chelonia state.
           'state/vuex/state', 'state/vuex/settings', 'state/vuex/commit', 'state/vuex/getters',
-          'chelonia/rootState', 'chelonia/contract/state', 'chelonia/contract/sync', 'chelonia/contract/isSyncing', 'chelonia/contract/remove', 'chelonia/contract/retain', 'chelonia/contract/release', 'controller/router',
+          'chelonia/rootState', 'chelonia/contract/state', 'chelonia/contract/isSyncing', 'chelonia/contract/remove', 'chelonia/contract/retain', 'chelonia/contract/release', 'controller/router',
           'chelonia/contract/suitableSigningKey', 'chelonia/contract/currentKeyIdByName',
           'chelonia/storeSecretKeys', 'chelonia/crypto/keyId',
           'chelonia/queueInvocation', 'chelonia/contract/wait',
@@ -497,14 +497,6 @@ async function startApp () {
             sbp('gi.ui/clearBanner')
             sbp('okTurtles.events/emit', ONLINE)
             console.info('reconnected to pubsub!')
-          },
-          'subscription-succeeded' (event) {
-            const { channelID } = event.detail
-            if (channelID in sbp('state/vuex/state').contracts) {
-              sbp('chelonia/contract/sync', channelID, { force: true }).catch(err => {
-                console.warn(`[chelonia] Syncing contract ${channelID} failed: ${err.message}`)
-              })
-            }
           }
         })
       })
@@ -543,7 +535,7 @@ async function startApp () {
         // it is important we first login before syncing any contracts here since that will load the
         // state and the contract sideEffects will sometimes need that state, e.g. loggedIn.identityContractID
         await sbp('gi.app/identity/login', { identityContractID })
-        await sbp('chelonia/contract/sync', identityContractID, { force: true })
+        await sbp('chelonia/contract/sync', identityContractID)
         const contractIDs = groupContractsByType(cheloniaState.contracts)
         await syncContractsInOrder(contractIDs)
       }).catch(async e => {

frontend/views/pages/PendingApproval.vue

@@ -54,8 +54,8 @@ export default ({
   },
   mounted () {
     this.ephemeral.groupIdWhenMounted = this.currentGroupId
-    sbp('chelonia/contract/wait', this.ourIdentityContractId).then(async () => {
-      await sbp('chelonia/contract/sync', this.ephemeral.groupIdWhenMounted)
+    this.ephemeral.syncPromise = sbp('chelonia/contract/wait', this.ourIdentityContractId).then(async () => {
+      await sbp('chelonia/contract/retain', this.ephemeral.groupIdWhenMounted, { ephemeral: true })
       this.ephemeral.contractFinishedSyncing = true
       if (this.haveActiveGroupProfile) {
         this.ephemeral.groupJoined = true
@@ -64,6 +64,13 @@ export default ({
       console.error('[PendingApproval.vue]: Error waiting for contract to finish syncing', e)
     })
   },
+  beforeDestroy () {
+    this.ephemeral.syncPromise?.then(() => {
+      sbp('chelonia/contract/release', this.ephemeral.groupIdWhenMounted, { ephemeral: true }).catch(e => {
+        console.error('[PendingApproval.vue]: Error releasing group contract', e)
+      })
+    })
+  },
   watch: {
     groupState (to) {
       if (to?.settings && this.ephemeral.settings !== to.settings) {

shared/domains/chelonia/chelonia.js

@@ -558,6 +558,20 @@ export default (sbp('sbp/selectors/register', {
     sbp('chelonia/private/startClockSync')
     this.pubsub = createClient(pubsubURL, {
       ...this.config.connectionOptions,
+      handlers: {
+        'subscription-succeeded': (event) => {
+          const { channelID } = event.detail
+          if (this.subscriptionSet.has(channelID)) {
+            // For new subscriptions, some messages could have been lost
+            // between the time the subscription was requested and it was
+            // actually set up. In these cases, force sync contracts to get them
+            // updated.
+            sbp('chelonia/private/out/sync', channelID, { force: true }).catch(err => {
+              console.warn(`[chelonia] Syncing contract ${channelID} failed: ${err.message}`)
+            })
+          }
+        }
+      },
       // Map message handlers to transparently handle encryption and signatures
       messageHandlers: {
         ...(Object.fromEntries(
@@ -609,7 +623,7 @@ export default (sbp('sbp/selectors/register', {
         )),
         [NOTIFICATION_TYPE.ENTRY] (msg) {
           // We MUST use 'chelonia/private/in/enqueueHandleEvent' to ensure handleEvent()
-          // is called AFTER any currently-running calls to 'chelonia/contract/sync'
+          // is called AFTER any currently-running calls to 'chelonia/private/out/sync'
           // to prevent gi.db from throwing "bad previousHEAD" errors.
           // Calling via SBP also makes it simple to implement 'test/backend.js'
           const { contractID } = GIMessage.deserializeHEAD(msg.data)
@@ -796,28 +810,26 @@ export default (sbp('sbp/selectors/register', {
   },
   // 'chelonia/contract' - selectors related to injecting remote data and monitoring contracts
   // TODO: add an optional parameter to "retain" the contract (see #828)
-  'chelonia/contract/sync': function (contractIDs: string | string[], params?: { force?: boolean, resync?: boolean }): Promise<*> {
+  // eslint-disable-next-line require-await
+  'chelonia/contract/sync': async function (contractIDs: string | string[], params?: { resync?: boolean }): Promise<*> {
+    // The exposed `chelonia/contract/sync` selector is meant for users of
+    // Chelonia and not for internal use within Chelonia.
+    // It should only be called after `/retain` where needed (for example, when
+    // starting up Chelonia with a saved state)
     const listOfIds = typeof contractIDs === 'string' ? [contractIDs] : contractIDs
-    const forcedSync = !!params?.force
-    return Promise.all(listOfIds.map(contractID => {
-      if (!forcedSync && this.subscriptionSet.has(contractID)) {
-        const rootState = sbp(this.config.stateSelector)
-        if (!rootState[contractID]?._volatile?.dirty) {
-          return sbp('chelonia/private/queueEvent', contractID, ['chelonia/private/noop'])
+    // Verify that there's a valid reference count
+    listOfIds.forEach((id) => {
+      if (checkCanBeGarbageCollected.call(this, id)) {
+        if (process.env.CI) {
+          Promise.reject(new Error('[chelonia] Missing reference count for contract ' + id))
         }
+        console.error('[chelonia] Missing reference count for contract ' + id)
+        throw new Error('Missing reference count for contract')
       }
-      // enqueue this invocation in a serial queue to ensure
-      // handleEvent does not get called on contractID while it's syncing,
-      // but after it's finished. This is used in tandem with
-      // queuing the 'chelonia/private/in/handleEvent' selector, defined below.
-      // This prevents handleEvent getting called with the wrong previousHEAD for an event.
-      return sbp('chelonia/private/queueEvent', contractID, [
-        'chelonia/private/in/syncContract', contractID, params
-      ]).catch((err) => {
-        console.error(`[chelonia] failed to sync ${contractID}:`, err)
-        throw err // re-throw the error
-      })
-    }))
+    })
+    // Call the internal sync selector. `force` is always true as using `/sync`
+    // besides internally is only needed to force sync a contract
+    return sbp('chelonia/private/out/sync', listOfIds, { ...params, force: true })
   },
   'chelonia/contract/isSyncing': function (contractID: string, { firstSync = false } = {}): boolean {
     const isSyncing = !!this.currentSyncs[contractID]
@@ -891,7 +903,7 @@ export default (sbp('sbp/selectors/register', {
         }
       })
     }
-    return await sbp('chelonia/contract/sync', listOfIds)
+    return await sbp('chelonia/private/out/sync', listOfIds)
   },
   // the `try` parameter does not affect (ephemeral or persistent) reference
   // counts, but rather removes a contract if the reference count is zero
@@ -1107,7 +1119,7 @@ export default (sbp('sbp/selectors/register', {
       prepublish: hooks.prepublishContract,
       postpublish: hooks.postpublishContract
     })
-    await sbp('chelonia/contract/sync', contractID)
+    await sbp('chelonia/private/out/sync', contractID)
     const msg = await sbp(actionEncryptionKeyId
       ? 'chelonia/out/actionEncrypted'
       : 'chelonia/out/actionUnencrypted', {