Clarify use of sync. Closes #2298 (#2303)

* Clarify use of sync. Closes #2298

* Feedback

* Improve comment

* Debug

* Remove debug statements

* Add comment and fix selector name

docs/Information-Flow.md

@@ -70,25 +70,118 @@ In our example above, the name of the *contract action* generated by `sbp('gi.ac
 
 ##### Subscribing to a contract
 
-Subscribing to a contract and syncing all of its updates is done by calling `'chelonia/contract/sync'`:
+Chelonia implements _reference counting_ to automatically manage contract
+subscriptions. When the reference count is positive, a contract subscription
+is created, meaning that the contract will be synced and Chelonia will listen
+for new updates. When the reference count drops to zero, the contract will be
+automatically removed(*).
+
+Subscribing to a contract and syncing all of its updates is done by calling `'chelonia/contract/retain'`:
 
 For example:
 
 ```js
-sbp('chelonia/contract/sync', contractID)
+sbp('chelonia/contract/retain', contractID)
 
 // OR wait until it finishes syncing:
 
-await sbp('chelonia/contract/sync', contractID)
+await sbp('chelonia/contract/retain', contractID)
 ```
 
 This will subscribe us to the contract and begin listening for new updates.
 
+When the contract is no longer needed, you can call `'chelonia/contract/release'`.
+
+For example:
+
+```js
+await sbp('chelonia/contract/release', contractID)
+```
+
+###### When to call `retain` and `release`
+
+Normally, you should call `retain` each time an event is about to happen that
+requires receiving updates for a contract. You should then call `release` when
+that reason for subscribing no longer holds.
+
+**IMPORTANT:** Each call to `release` must have a corresponding call to `retain`.
+In other words, the reference count cannot be negative.
+
+Three examples of this are:
+
+  * **Writing to a contract.** Writing to a contract requires being subscribed
+    to it, so you should call `retain` before sending an action to it. The
+    contract can be released by calling `release` after the writes have completed.
+  * **Subscribing to related contracts in side-effects.** A common use case for
+    calling `retain` and `release` is when a contract is related to other
+    contracts. For example, you could have users that can be members of different
+    groups. Every time a user joins a group, you would call `retain` in the
+    side-effect of that action, and then call `release` when the group membership
+    ends.
+  * **Logging a user in.** If your application has users that are represented by
+    contracts, those contracts usually need to be subscribed during the life of
+    a session. This would be an example where you would have a call to `retain`
+    when the account is created and then there could be no `release` call.
+
+###### Ephemeral reference counts
+
+Chelonia maintains two different reference counts that you can directly control
+using `retain` and `release`: ephemeral and non-ephemeral.
+
+Non-ephemeral reference counts are stored in the root Chelonia state and are
+meant to be persisted. In the examples above, the examples of subscribing to
+related contracts and logging a user in would fall in this category. If Chelonia
+is restarted, you want those references to have the same value.
+
+On the other hand, _ephemeral_ reference counts work a differently. Those are
+only stored in RAM and are meant to be used when restarting Chelonia should
+_not_ restore those counts. The example of writing to a contract would be one
+of those cases. If Chelonia is restarted (for example, because a user refreshes
+the page) and you're in the middle of writing to a contract, you would not want
+that reference to persist because you'd have no way of knowing you have to call
+`release` afterwards, which would have the effect of that contract never being
+removed.
+
+All calls to `retain` and `release` use non-ephemeral references by default. To
+use ephemeral references, pass an object with `{ ephemeral: true }` as the last
+argument. Note that ephemeral `retain`s are paired with ephemeral `release`s,
+and non-ephemeral `retain`s are paired with non-ephemeral `release`s (i.e.,
+don't mix them).
+
+For example,
+
+```js
+// NOTE: `retain` must be _outside_ of the `try` block and immediately followed
+// by it. This ensures that if `release` is called if and only if `retain`
+// succeeds.
+await sbp('chelonia/contract/retain', contractID, { ephemeral: true })
+try {
+  // do something
+} finally {
+  await sbp('chelonia/contract/release', contractID, { ephemeral: true })
+}
+```
+
+###### `chelonia/contract/sync`
+
+In addition to `retain` and `release`, there is another selector that's
+relevant: `chelonia/contract/sync`. You use `sync` to force fetch the latest
+state from the server, or to create a subscription if there is none (this is
+useful when bootstrapping your app: you already have a state, but Chelonia
+doesn't know you should be subscribed to a contract). `sync` doesn't affect
+reference counts and you should always call `sync` on contracts that have at
+least one refeence. This means that you need to, at some point, have called
+`retain` on that contract first.
+
 (...WIP...)
 
 When subscribed to a Contract, the user is updated each time an action there is called, even if the action wasn't triggered by the user itself. (TODO: Add link/reference to where this happens)
 
 So you don't need to worry about this for now, it just works 🔮.
 
+(*) The actual mechanism is more involved than this, as there are some other
+reasons to listen for contract updates. For example, if contracts use foreign
+keys (meaning keys that are defined in other contracts), Chelonia may listen for
+events in those other contracts to keep keys in sync.
 
-That's all for now! Feel free to dive even more deeply in the files mentioned so far and complement these docs with your discoveries.
+That's all for now! Feel free to dive even more deeply in the files mentioned so far and complement these docs with your discoveries.

frontend/controller/actions/identity.js

@@ -171,35 +171,39 @@ export default (sbp('sbp/selectors/register', {
         hooks: {
           postpublishContract: async (message) => {
             // We need to get the contract state
-            await sbp('chelonia/contract/sync', message.contractID())
-
-            // Register password salt
-            const res = await fetch(`${sbp('okTurtles.data/get', 'API_URL')}/zkpp/register/${encodeURIComponent(username)}`, {
-              method: 'POST',
-              headers: {
-                'authorization': await sbp('chelonia/shelterAuthorizationHeader', message.contractID()),
-                'content-type': 'application/x-www-form-urlencoded'
-              },
-              body: new URLSearchParams({
-                'r': r,
-                's': s,
-                'sig': sig,
-                'Eh': Eh
+            await sbp('chelonia/contract/retain', message.contractID(), { ephemeral: true })
+
+            try {
+              // Register password salt
+              const res = await fetch(`${sbp('okTurtles.data/get', 'API_URL')}/zkpp/register/${encodeURIComponent(username)}`, {
+                method: 'POST',
+                headers: {
+                  'authorization': await sbp('chelonia/shelterAuthorizationHeader', message.contractID()),
+                  'content-type': 'application/x-www-form-urlencoded'
+                },
+                body: new URLSearchParams({
+                  'r': r,
+                  's': s,
+                  'sig': sig,
+                  'Eh': Eh
+                })
               })
-            })
 
-            if (!res.ok) {
-              throw new Error('Unable to register hash')
-            }
+              if (!res.ok) {
+                throw new Error('Unable to register hash')
+              }
 
-            userID = message.contractID()
-            if (picture) {
-              try {
-                finalPicture = await imageUpload(picture, { billableContractID: userID })
-              } catch (e) {
-                console.error('actions/identity.js picture upload error:', e)
-                throw new GIErrorUIRuntimeError(L('Failed to upload the profile picture. {codeError}', { codeError: e.message }), { cause: e })
+              userID = message.contractID()
+              if (picture) {
+                try {
+                  finalPicture = await imageUpload(picture, { billableContractID: userID })
+                } catch (e) {
+                  console.error('actions/identity.js picture upload error:', e)
+                  throw new GIErrorUIRuntimeError(L('Failed to upload the profile picture. {codeError}', { codeError: e.message }), { cause: e })
+                }
               }
+            } finally {
+              await sbp('chelonia/contract/release', message.contractID(), { ephemeral: true })
             }
           }
         },
@@ -254,10 +258,10 @@ export default (sbp('sbp/selectors/register', {
         } else {
         // If there is a state, we've already retained the identity contract
         // but might need to fetch the latest events
-          await sbp('chelonia/contract/sync', identityContractID, { force: true })
+          await sbp('chelonia/contract/sync', identityContractID)
         }
       } catch (e) {
-        console.error('Error during login contract sync', e)
+        console.error('[gi.actions/identity] Error during login contract sync', e)
         throw new GIErrorUIRuntimeError(L('Error during login contract sync'), { cause: e })
       }
 
@@ -364,7 +368,6 @@ export default (sbp('sbp/selectors/register', {
       //      queues), including their side-effects (the `${contractID}` queues)
       //   4. (In reset handler) Outgoing actions from side-effects (again, in
       //      the `encrypted-action` queue)
-      cheloniaState = await sbp('chelonia/rootState')
       await sbp('okTurtles.eventQueue/queueEvent', 'encrypted-action', () => {})
       // reset will wait until we have processed any remaining actions
       cheloniaState = await sbp('chelonia/reset', async () => {

frontend/controller/actions/index.js

@@ -208,10 +208,5 @@ sbp('sbp/selectors/register', {
         }
       })
     }
-
-    // TODO: Temporary sync until the server does signature validation
-    // This prevents us from sending messages signed with just-revoked keys
-    // Once the server enforces signatures, this can be removed
-    await sbp('chelonia/contract/sync', contractID, { force: true })
   }
 })

frontend/controller/actions/utils.js

@@ -309,8 +309,19 @@ export async function createInvite ({ contractID, quantity = 1, creatorID, expir
 export function groupContractsByType (contracts: Object): Object {
   const contractIDs = Object.create(null)
   if (contracts) {
+    // Note: `references` holds non-ephemeral references (i.e., explicit
+    // calls to `retain` without `{ ephemeral: true }`). These are the contracts
+    // that we want to restore.
+    // Apart from non-ephemeral references, `references` may not be set for
+    // contracts being 'watched' for foreign keys. The latter are managed
+    // directly by Chelonia, so we also don't subscribe to them
     // $FlowFixMe[incompatible-use]
-    Object.entries(contracts).forEach(([id, { type }]) => {
+    Object.entries(contracts).forEach(([id, { references, type }]) => {
+      // If the contract wasn't explicitly retained, skip it
+      // NB! Ignoring `references` could result in an exception being thrown, as
+      // as `sync` may only be called on contracts for which a reference count
+      // exists.
+      if (!references) return
       if (!contractIDs[type]) {
         contractIDs[type] = []
       }
@@ -343,7 +354,7 @@ export async function syncContractsInOrder (groupedContractIDs: Object): Promise
       // Sync contracts in order based on type
       return getContractSyncPriority(a) - getContractSyncPriority(b)
     }).map(([, ids]) => {
-      return sbp('chelonia/contract/sync', ids, { force: true })
+      return sbp('chelonia/contract/sync', ids)
     }))
   } catch (err) {
     console.error('Error during contract sync (syncing all contractIDs)', err)

frontend/controller/app/identity.js

@@ -347,7 +347,7 @@ export default (sbp('sbp/selectors/register', {
           sbp('okTurtles.events/off', LOGIN_ERROR, loginErrorHandler)
 
           const errMessage = e?.message || String(e)
-          console.error('Error during login contract sync', e)
+          console.error('[gi.app/identity] Error during login contract sync', e)
 
           const promptOptions = {
             heading: L('Login error'),

frontend/setupChelonia.js

@@ -270,7 +270,7 @@ const setupChelonia = async (): Promise<*> => {
       'subscription-succeeded' (event) {
         const { channelID } = event.detail
         if (channelID in sbp('chelonia/rootState').contracts) {
-          sbp('chelonia/contract/sync', channelID, { force: true }).catch(err => {
+          sbp('chelonia/contract/sync', channelID).catch(err => {
             console.warn(`[chelonia] Syncing contract ${channelID} failed: ${err.message}`)
           })
         }
@@ -280,12 +280,12 @@ const setupChelonia = async (): Promise<*> => {
 
   await sbp('gi.db/settings/load', SETTING_CURRENT_USER).then(async (identityContractID) => {
     // This loads CHELONIA_STATE when _not_ running as a service worker
-    const cheloniaState = await sbp('gi.db/settings/load', SETTING_CHELONIA_STATE)
+    const cheloniaState = await sbp('chelonia/rootState')
     if (!cheloniaState || !identityContractID) return
     if (cheloniaState.loggedIn?.identityContractID !== identityContractID) return
     // it is important we first login before syncing any contracts here since that will load the
     // state and the contract sideEffects will sometimes need that state, e.g. loggedIn.identityContractID
-    await sbp('chelonia/contract/sync', identityContractID, { force: true })
+    await sbp('chelonia/contract/sync', identityContractID)
     const contractIDs = groupContractsByType(cheloniaState.contracts)
     await syncContractsInOrder(contractIDs)
   })

frontend/views/pages/PendingApproval.vue

@@ -15,11 +15,12 @@ div
 </template>
 
 <script>
-import sbp from '@sbp/sbp'
 import GroupWelcome from '@components/GroupWelcome.vue'
 import { PROFILE_STATUS } from '@model/contracts/shared/constants'
+import sbp from '@sbp/sbp'
 import SvgInvitation from '@svgs/invitation.svg'
 import { mapGetters, mapState } from 'vuex'
+import { CHELONIA_RESET } from '~/shared/domains/chelonia/events.js'
 
 export default ({
   name: 'PendingApproval',
@@ -54,15 +55,43 @@ export default ({
   },
   mounted () {
     this.ephemeral.groupIdWhenMounted = this.currentGroupId
-    sbp('chelonia/contract/wait', this.ourIdentityContractId).then(async () => {
-      await sbp('chelonia/contract/sync', this.ephemeral.groupIdWhenMounted)
+    let reset = false
+    let destroyed = false
+
+    const syncPromise = sbp('chelonia/contract/wait', this.ourIdentityContractId).then(async () => {
+      if (destroyed) return
+      reset = false
+      // We don't want to accidentally unsubscribe from the group while this
+      // page is rendered, so we increase the ephemeral reference count.
+      // When this page is destroyed, the reference count is decreased as well.
+      // Proper (non-ephemeral) references are handled by the `identity/joinGroup`
+      // side-effects. In general, UI elements should not be changing
+      // non-ephemeral references.
+      await sbp('chelonia/contract/retain', this.ephemeral.groupIdWhenMounted, { ephemeral: true })
       this.ephemeral.contractFinishedSyncing = true
       if (this.haveActiveGroupProfile) {
         this.ephemeral.groupJoined = true
       }
     }).catch(e => {
       console.error('[PendingApproval.vue]: Error waiting for contract to finish syncing', e)
     })
+    const listener = () => { reset = true }
+    this.ephemeral.ondestroy = () => {
+      destroyed = true
+      sbp('okTurtles.events/off', CHELONIA_RESET, listener)
+      syncPromise.finally(() => {
+        if (reset) return
+        sbp('chelonia/contract/release', this.ephemeral.groupIdWhenMounted, { ephemeral: true }).catch(e => {
+          console.error('[PendingApproval.vue]: Error releasing contract', e)
+        })
+      })
+    }
+    // If Chelonia was reset, it means that ephemeral references have been
+    // lost, and we should not release the contract.
+    sbp('okTurtles.events/on', CHELONIA_RESET, listener)
+  },
+  beforeDestroy () {
+    this.ephemeral.ondestroy?.()
   },
   watch: {
     groupState (to) {