Further plans for CLI, @joystream/js, @joystream/types, cd-schemas

[Lezek123]

CLI:

Short term:

media

• Create/update video - allow providing some missing data: userDefinedLicense (path to file?), publishedBeforeJoystream, skippableIntroDuration
• Allow removing video (will involve removing multiple entities like VideoMedia, VideoMediaLocation etc)
• Allow removing a channel

content-directory

• Add a command to run content directory initialization script as Curator Lead
• Removing entities
• Lower-level commands for creating and managing entities, entity creation vouchers etc. (related issue: CLI: Managing entities (creating, updating, removing etc.) #1441)

working-groups

• Simplify providing input for working-groups:createOpening
  • It's quite hard to use currently, especially that most of the input isn't validated until the very end and cannot be easily changed if invalid (in that case user may be forced to start over, which can become quite frustrating)
  • We can now use JsonSchemaPrompt class to prompt for human_readable_text (so that we can avoid having to convert it to Struct) and validate it in real-time

api

• Create api:tx command allowing the user to send any extrinsic and provide the input through CLI prompts (or json, see: Scriptability section below)
  This was planned for a long time and it shouldn't take too much work now that we have a mechanism to prompt for almost any extrinsic input (it is used currently ie. by working-groups:createOpening command)
• api:tx / api:inspect - allow easier navigation between existing modules/methods etc (displaying a choice list in case they are not provided via a flag should be relatively easy)

Scriptability (related issue: #1571):

• Allow providing input as flags/args/json where this is not yet implemented (some commands only provide interactive interface, ie. working-groups:createOpening, account:choose)
• Give user the ability to choose between more human-readable output (ie. a table) and output that can be redirected to a file / other commands (ie. raw json, created entity id etc.). Some OCLIF funcionality can probably help facilitate this (ie. --no-color flag can be used to strip any custom colors from the output, cli.table can be converted to csv by passing an additional option etc.)

Other:

• Use the new api augmentations
  • Allows us the use @joystream/js as tsconfig path for better IDE navigation (also mentioned in @joystream/js section)
  • Allows more typesafe use of methods like buildAndSendExtrinsic (will give us compiler errors in case methods no longer exist etc.)

cd-schemas

Short term

• Since we need to publish it, we should probably change the name (to ie. @joystram/content)

Longer term

• Consider merging it with @joystream/js so that we can, for example, include methods like fetchAndParseEntity (currently part of CLI) as part of contentDirectoryTransport and generally be able to include content-directory related utilities in @joystream/js without the risk of creating circular references (and avoid having to publish/manage those two libraries separately).
• Support for vectors in property types
• Support for multiple schemas per class
• Find a better way to handle the links between json-schemas and json inputs in IDE (avoid having to manually change the vscode settings)

Monorepo setup:

This is related to libraries like cd-schemas and @joystream/js that should be by definition available for multiple other projects in the monorepo

• Make separate tsconfigs for IDE (tsconfig.json) and builds (tsconfig.build.json) of our TypeScript projects in the monorepo, so that they can share paths and reference each other more easily (we would then have the right handlig of jump-to-definition in IDE and it should also help us avoid re-building the libraries in order for the IDE to become aware of the changes etc.). Some research still needed w.r.t. how this would affect tools like ts-node etc.

Related sources: https://medium.com/@NiGhTTraX/how-to-set-up-a-typescript-monorepo-with-lerna-c6acda7d4559, https://medium.com/@NiGhTTraX/making-typescript-monorepos-play-nice-with-other-tools-a8d197fdc680

• Add publishing tests to CI. Perhaps lerna could offer some help here. npm pack can give us information about what will end up in the published package, but by itself it doesn't do anything to check if the package will work at all. We can combine it with running npm install and node index.js inside the extracted tarball, but this won't work if we have multiple packages that depend on each other and we want to publish them all at once. Perhaps we could have a CI workflow that would make tarballs of the packages that we plan to publish in a correct order (@joystream/types => @joystream/js => @joystream/cli) and modify the references in package.json files to point to tarballs, instead of not-yet published versions of our packages so that we can then run a check like npm install && node index.js inside the extracted tarballs succesfully (but probably there exsits a better way to handle this)

@joystream/js:

Some of this already mentioned in #1396:

• Come up with a more universal directory structure (partially described in Initial @joystream/js library #1572 as part of potential plan to move cd-schemas into @joystream/js) as there seem to be some functionality/assets that we may want to include there and that will be hard to fit into one of the existing categories (ie. json schemas)
• Documentation tooling (http://typedoc.org/)
• Use it to replace joy-roles transport in Pioneer (this will probably take a little more time that expected, so it may turn out not to be worth it)
• Include other common reusable functionality:
  • Api initialization (that would allow us to introduce some global api customization w.r.t. ie. handling historical queries, adding type namespaces etc.)
  • ExtrinsicHelper from cd-schemas (+ some methods that would allow us to retrieve data from expected events after sending an extrinsic, like the id from EntityCreated)
  • TBD

@joystream/types

Optimally we would have a single source of truth (on the frontend) about the current runtime types and have all the required TypeScript interfaces, augmentations etc. auto-generated from it with tools like @polkadot/typgen.

Json definitions seem like the most common standard for providing type definitions for a Substrate chain. This is a consistent format that we can use with @polkadot/api, @polkadot/typegen and other tools that would be built with supporting multiple chains in mind (ie. Hydra).

We used to have only custom class/codec types that couldn't be simply converted to json definitions. We can now convert them to json definitions with a custom script and then generate TypeScript api augmentations (with @polkadot/typegen) based on those definitions. We then have another script that overrides those augmentations so that they use our custom class types instead of interfaces generated by @polkadot/typegen, since we still use those class/codec types during api initialization in almost all our projects.

There are a few issues preventing a smooth migration to json types (+ autogenerated interfaces) only:

• Some of our projects still rely on custom methods (ie. ContentId.encode) or getters (ie. ModerationAction.rationale returning string instead of Text) that we have defined in our custom class-types. This has been normalized quite a bit now and with versioned-store types no longer needed and media app beeing removed from Pioneer this shouldn't be that much of an issue anymore. Methods like ContentId.decode could become part of @joystream/js or just be included in @joystream/types, but as separate functions that take the type (ie. ContentId) as argument.
• Some types/operations would be harder to represent if we would only have to rely on interfaces generated by @polkadot/typegen. For example, we cannot easily derive string union type for enum keys based on enum interface generated by the tool. We also wouldn't be able to use constructs like isOfType(someVariable) anymore. This could perhaps be solved by implementing a more customized tool based on @polkadot/typegen and/or by refactorization of some of our codebase.
• Types like BTreeSet or BTreeMap require some specific encoding (ie. BTreeSet values and BTreeMap keys need to be sorted in ascending order), which currently does not seem to be supported in the implementation of those types in @polkadot/types. We can partially solve this issue with our custom wrappers for those types, which we currently use in some cases (although we cannot, for example, "hook" into encoding of BTreeSet if it's an extrinsc arg and doesn't have any wrapper on the runtime)
• In the json defintions the types of Struct properties and Enum variants are provided as strings (ie. 'Membership', 'InputPropertyValue' etc.), so it would be slightly harder to navigate between them when adding new types (until the new interfaces are generated)

On the other side the benefits of the migration are:

• Way better clarity of @joystream/types library content - we would only have one set of types in json format and interfaces generated for them - no additional classes, no two different kinds of api augmentations etc. (the directory structure could match the one in: https://github.com/polkadot-js/docs/tree/master/docs/api/examples/promise/typegen/src)
• It is quite easy to modify types in json format and there aren't that many rules that need to be followed (like the ones that need to be followed now w.r.t. how we export classes in order to be able run all the generation scripts without running into any issues),
• We would have support of all the newset tools from the @polkadot/api family and other tools/projects built with Substrate chains in mind, that would expect us to use json type definitions

Currently possible improvments

Currently the costs of having types only in the json-format seem to still outweigh the benefits, so for now we should probably settle on using class-types and augment-codec augmentations in our monorepo TypeScript projects (until there comes a time we are forced to switch or it just becomes a way better deal due to some improvements in @polkadot/api and/or some related tooling).

There are still a few more steps we can take now to make sure the library is in good shape and more ready for this possible "transformation" in the future. This could involve:

• Getting rid of custom getters that change the original field type (ie. ModerationAction.rationale) - they are the most likely source of some potential incompatibility issues between types registered from json defintions and class/codec types.
• Creating utility types like EnumKey<T> (we can derive them from JoyEnum type) to avoid having to define multiple additional separate types for enums (and sidestep the problem of inconsistent naming etc.)
• Add a documentation that would clearly explain the difference between codec/class and json types, types/augment and types/augment-codec etc. (I already made an attempt to describe it a few times, ie. in @joystream/types tooling improvements - augment-codec and defs.json #1460, @joystream/types - api augmentation #1177 etc.)
• Finding the most universal approach to solve BTreeSet / BTreeMap issues