Babylon: Create initial, consumable joystream-js library

[Lezek123]

During the last few weeks/months I was aiming to extract some of the potentially reusable functionality from Pioneer's packages to pioneer/packages/src/joy-utils. This process began with https://github.com/Joystream/apps/pull/465 and the main idea is also further described in that PR.

I think Babylon release may be a good time to extract most of the code in joy-utils into the first version of long-awaited joystream-js library.

Benefits

Benefit 1: Transports

Currently the main benefit of doing this is that it will make the so-called "transports" (see: https://github.com/Joystream/joystream/tree/iznik/pioneer/packages/joy-utils/src/transport) accessible for all our JavaScript projects (Pioneer, CLI, storage node, status server etc.).

Basically all of those projects currently have some custom wrapper class/classes for the api to simplify fetching, combining and parsing commonly used data. Having this normalized across projects could greatly simplify adapting them after runtime changes that affect the api/types (which currently generates a need to update the same / similar query scenarios in all projects separately).

Some examples of data querying scenarios and how they are simplified via transports:

• Fetching all of given working group opening applications along with applicants' profiles and their stakes.
  This currently requires executing tons of separate queries, talking to lots of different runtime modules and parsing a lot of the data, but using the existing transport we can just do it with: const applications = await transport.workingGroups.openingApplications(group, openingId)
• Fetching the entire proposal discussion thread along with all the posts and involved users' profiles.
  This requires fetching the related thread, then executing some still relatively complex logic of fetching all double map entries by first key, then fetching member profiles, combining results, parsing dates etc. With transport we just do: const discussion = await transport.proposals.discussion(proposalId)
• Fetching a batch of proposals filtered by status, containing data like: proposal details, proposal type (comes from ProposalDetails enum), approval/slashing quorum, proposer profile etc. (it also contains information that allows easy pagination of the results, like total number of proposals that match the filter)
  Available via transport.proposals.proposalsBatch(status, batchNumber, batchSize). Some of this data we can't even query from the chain, that's where joy-utils/consts kick in (see below in Common constants)

There are lots of other similar common scenarios that are already covered by the transport classes inside joy-utils (some of them very simple, like querying timestamp based on block number)

Other benefits that shared transports provide:

• Some common query helper methods like entriesByIds, ie.: const allWorkers = entriesByIds<WorkerId, Worker>(api.query.storageWorkingGroup.workerById); (normally that would take a few more lines of code)
• Ability to implement global caching or create a backend api that will cache the results for multiple projects (similar to query node). Currently ApiQueryCache is used by default, which helps in situations when the same query is executed multiple times in context of the same (current) block
• Ability to normalize the format of some data objects across projects, which may be a good thing to have before query node becomes fully functional (it may simplify the migration). It also allows us to normalize data mocking (we can create data mocking layer that will work across projects)

Benefit 2: Common constants

There is still some data that isn't exposed by the chain, but can be very useful (or even required) by some apps. Some of it is already hardcoded inside: https://github.com/Joystream/joystream/blob/iznik/pioneer/packages/joy-utils/src/consts

This includes data like:

• Proposals required stakes, approval/slashing quorum, categories etc. (grouped by proposal type)
• Api methods to fetch voting/grace periods grouped by proposal type
• Names of working group api modules grouped by working group

Benefit 3: Types and typescript helpers

All the types used by the library (ie. interfaces for objects returned by the transport etc.) are available inside: https://github.com/Joystream/joystream/tree/iznik/pioneer/packages/joy-utils/src/types.

In the future I see the potential of also creating generic utility types that can be used by other projects, ie. SimplifiedTypeInterface from https://github.com/Joystream/joystream/blob/iznik/pioneer/packages/joy-utils/src/types/common.ts can be extended to allow converting complex structs like OpeningPolicyCommitment into simple, but still type-safe objects that will be accepted and correctly converted into types expected by the runtime via @polkadot/api (it already does that, but doesn't provide the TS interface)

Benefit 4: Other utility functions

joy-utils currently also contain a few other, generic utility functions in https://github.com/Joystream/joystream/tree/iznik/pioneer/packages/joy-utils/src/functions, ie.:

• A method to format reward (RewardRelationship / RewardPolicy) as string
• Some helpers related to account/keys (https://github.com/Joystream/joystream/blob/iznik/pioneer/packages/joy-utils/src/functions/accounts.ts)
• Some more generic functions that, for example, simplify preserving TypeSciprt compatibility with strict eslint rules (https://github.com/Joystream/joystream/blob/iznik/pioneer/packages/joy-utils/src/functions/misc.ts)

Benefit 5: A place where we can keep shared tooling, validation, schemas, scripts etc.

There is currently no obvious place to keep reusable scripts, json schemas etc. that can be useful for all of our JavaScript projects.

Once we have joystream-js library we can expand it to contain as much of the reusable code as possible (or later split into a few more specific libraries) in order to keep it in one place and reduce the amout of work needed to update our JavaScript projects one-by-one.

We can also for example:

• Normalize common processes (like api initialization)
• Provide an easy way to perform more complex tasks like querying old chain state (Reading old chain state + example usage #1092) etc. (we can then even integrate it into the transports too)
• Simplify processes like channel creation, video upload etc., so they can be easily integrated into multiple different tools (ie. CLI, Pioneer, Atlas) depending on the need.

To sum it up, I think there is a lot of future potential beyond what's currently available in joy-utils and some of it can be leveraged even during the current release (ie. the introduction of new content directory schemas)

Implementation

For this release my main goal is to factor joystream-js out of joy-utils and make is publishable and consumable as a standalone library. joy-utils package is currently still dependent on Pioneer and the monorepo setup and it's hard for other projects to take the benefit of it.

Babylon release also seems like a good time to normalize workingGroups transport. The main blocker to do this before was that joy-roles package in Pioneer needed to handle both contentWorkingGroup and storageWorkingGroup (which currently have quite different logic), while CLI and proposals system in Pioneer only needed to deal with storageWorkingGroup (which is part of the updated workingGroup runtime module). Now that this will be normalized, it makes sense to also have one "transport-layer" shared between Pioneer (joy-roles, joy-proposals), CLI and possibly status server.

The last benefit of having joystream-js during Babylon is that we can use it as a place to keep new content directory schemas and possibly some functionality related to those (ie. to make tasks like creating a channel, uploading a video etc. easier for other projects in the future)

Other projects

Having access to a consumable joysream-js library may be useful for projects like storage-node etc. too during this release. Some of the reusable code can also be moved to joysream-js from cli and storage-node (ie. transactions batching) to share the benefits of worked out solutions and move faster in the development, but as mnaamani once pointed out, this can be step-by-step process and I don't think it's feasible to do everything at once during this release.

Documentation

In the future we can use tools like http://typedoc.org/ to create a more clear documentation with examples etc. and provide an easy way for the developers to start building tools for Joystream. During this release I will try to create some very basic documentation to explain what's possible (as I already partially did in this issue)

Scope of this issue (Babylon)

• Factor joystream-js out of Pioneer's joy-utils - make it "publishable" and consumable
• Make CLI and Pioneer depend on joystream-js
• Normalize workingGroups transport between CLI and Pioneer
• Create a very basic README.md documentation for joystream-js (partially based on this issue description)

┆Issue is synchronized with this Asana task by Unito

[bedeho]

A few reactions

1. I suspect almost all the transport code related to reading will be displaced by the query node, and as such has no long term future unless we want to maintain a separate codebase for apps that try to be powered directly be a full node?
   This is not entirely without merit, in particular if light client validation can be added, as it means the query node trust problem is sidestepped. However, I don't think we can justify maintaining and adding additional such reading transport code without very specific goals for this, so I am sceptical.

2. I think the transport code does not do any writing/extrinsics?

3. The most important thing we have to settle is: what is the role of joystream.js?

Is joystream.js the canonical developer library when they want to write an application for the Joystream Network, or is it just an internal library? The name certainly seems to suggest the former, and if not, then I think we at least need to perhaps change the name in order to make space for a future application developer facing library.

Now, what do I mean by applications for the Joystream Network? By that I mean that it gives a very high level of abstraction for talking ot the network as a whole, for example in Atlas/Pioneer/CLI, so it will allow you to talk to query nodes, storage nodes, full nodes, add serving nodes and other future service infrastructure, through one unified library. It would allow you to generate keys for roles in a way that follows the Joystream wallet standard, so that you can recover all your role keys later with one seed, etc.

If we take this as the role of joystream.js, could it then at the same time be the library used by the internals of the system, like the storage or query node?

[Lezek123]

Issue covered partially in #1572 due to complexity of updating Pioneer's joy-roles app and the need for additional testing it would generate:

• Factor joystream-js out of Pioneer's joy-utils - make it "publishable" and consumable - Covered by Initial @joystream/js library #1572
• Make CLI and Pioneer depend on joystream-js - Covered by Initial @joystream/js library #1572
• Normalize workingGroups transport between CLI and Pioneer - Covered only for CLI and joy-proposals in Initial @joystream/js library #1572
• Create a very basic README.md documentation for joystream-js (partially based on this issue description) - moved to a separate issue: @joystream/js - README #1598