You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is a preliminary document, it must be checked by key contributors to give input on expected full costs of various intended changes, and also whether goals in one area are fully reflected in requirements of dependant areas.
Name
Sumer
Brand
Goals
1. Deploy metaprotocol content directory.
2. Publishing in Atlas.
Overview
Alert
In the end, make sure you understand this example to see how everything on the infrastructure+runtime side is brought together, even if this is not your primary focus area
This release is a synthesis of distinct strands of improvement which became most practical to plan and execute in one release due to the interdependencies involved:
A publishing UI for WYSIWYG publication and management of content.
A storage system storage system that allows the publishing experience we are building will be able reflect, as closely as possible, the properties of the storage system in the long term. In particular being able to do a single on-chain operation for simultaneously uploading an unlimited number of assets and editing the content directory.
A content directory query node API which is aware of upload latencies or interruptions for media assets.
A new content directory which resolves long standing tradeoffs.
In what follows we will dive more deeply into each release goal, which then touches upon the
Metaprotocol content directory
The currently deployed content directory, described here
it is extremely complex to reason about, both for maintainers, but also for everyone else trying to work with it.
it is inflexible, in that it allowed you to do certain things, but if you wanted something different, like a DAO owning a channel, you just had to add even more complexity and special cases.
the complexity comes from two sources
having the chain validate the structure of metadata
modelling a generalised permission model for access control and resource usage
neither of these actually made much sense in the full context of what we are doing. For 1, higher levels of the stack, like the query node and apps, end up having too validation to generate convenient statically type safe interfaces anyway, and for 2 it is just too complex yet does not allow us to do new things we come up with, so its a bad tradeoff. Therefore we made a smart contract version of the content directory where metadata was not validated by the contract, this solves 1 and 2. The approach taken is described here
This solution was much better, but it became clear that even smart contracts that did not look at metadata were really a suboptimal half way house, and that a so called metaprotocol would be the best tradeoff available, see full description here
Implement the content directory using a metaprotocol, and there will be no dedicated content directory in the state of the chain.
Introduce an EVM smart contract module as part of Olympia, so that we can leverage existing code bases for DAOs and Defi, as well as unlock general permissionless community innovation. This integration work is however not part of this release.
Publishing in Atlas
Publishing Today
Current it is not possible to publish in atlas, in fact the only way to create and manage channels and videos is using a CLI, which is extremely cumbersome, as it requires learning about and correctly doing the following
Manually getting tokens and a membership
Downloadig & installing Node & NPM.
Installing CLI.
Importing membership account into CLI.
Manually preparing json metadata files and assets for a channel.
Doing command (hopefully works).
...then a bunch of magic steps like 5+6 this per video publication.
Publishing in Atlas
A full onboarding, membership creation, channel creation, dashboard, video publishing and management experience has been designed, see here
Importantly, a broader set of features have been designed than we should aspire to build in this release, so we are focusing on a core subset of features related to publishing, without accumulating tech debt. It was originally done under the following design brief,
which itself is not in synch with the final designed product, because we decided to change some of the constraints in process - most importantly having Joystream hosted media assets.
However, mistakes were made on the part of Bedeho when directing this effort, hence some mismatches have been identified between what the design presumes and the behaviour of the system, and some other more minor areas of improvement, see here
Add a simple scenario to integration tests for multi-upload.
Initial Division of labour
Shamil does everything.
Content Directory
Team
Mokhtar
Work Scope
Define a metaprotocol (messages + state) for the content directory which corresponds to the content and curation model in the current content directory. The current content directory model, as captured in the various schemas, may be bloated due to being modelled on top of the version+relational old content directory, review for opportunities to simplify and throw stuff out. The curator permission model obviosuly also needs to be absorbed into the model, but review for chances to simplify that also. Initial work with protocol buffers has been very promising for defining messages, and getting efficient serialization/deserialization code in every major language. The protocol description could easily be stated in terms of these messages. It may be the case that some versioning model could make sense, and protocol buffers have explicit support for this, but I suggest we stick to a simple model for now, given the number of moving parts in this release, and the historic unpredictability of what we do in our content layer. It would be highly desirable if part of the output here is a document that can go into the handbook, and which can be read by the query node developers who will write the query node input schema and implement the message processing.
Mirror new content directory developed, use early API drafts to unlock Atlas development.
Add account and membership awareness required to power the onboarding flow in Atlas.
(big task) Write new mappings for metaprotocol.
Current Status
Nothing is done here.
Initial Division of labour
Atlas Team does content directory part of input schema
Metin does membership and account side of input schema.
Split mapping work evenly among Metin and Arsen until its done.
Atlas
Team
Atlas Team
Francesco
Klaudiusz
Maciej
Bartosz
possible newcomers
Work Scope
As mentioned above there is a full design for the publisher side, however, it needs to be restricted in the following way
No channel dashboard.
Update or create faucet server for powering the onboarding flow.
Introduce light weight client side image clipping in the image editor prior to upload. Important: only apply operation when transaction initiation is done, because user may want to multiple times before comitting.
Develop library for interacting with content directory, using metaprotocol, and storage system which can be reused in Atlas, integration tests and CLI. There is already some library code developed for this used in existing CLI, but its not clear how useful it is to build on it given the deep changes. Also see this issue about getting results, it may be in scope. Metaprotocol DevUX #2018.
Initial Division of labour
Klaudiusz will allocate tasks internally in Atlas team.
CLI & Tooling
Team
Arsen
Mokhtar
Work Scope
Update CLI to work with new content directory
publishing
curation
Update mass-uploading script to populate using old seed content. There is no need to migrate over anything else, because no one is publishing with existing CLI solution.
Current Status
Nothing is done here.
Initial Division of labour
Unclear, perhaps Mokhtar focuses on CLI and Arsen on the mass upload script?
Network Testing
Team
Mokhtar
Metin
Arsen
Work Scope
Updated to work with runtime upgrade.
Does filtering in search, as above, allow to filter on union fields?
Current Status
Nothing has been done.
Initial Division of labour
Mokhtar does everything.
Deployment
Team
Mokhtar
Arsen
Work Scope
Status server
Telegram bots
Storage system
Polkadotjs library & examples
Current Status
Nothing has been done.
Initial Division of labour
Unclear.
Manual QA
Team
Martin
Ben
Work Scope
Write testing plan in advance and share with Bedeho.
Conduct test in accordance with plan. The testing here should be targeting CLI, Query Node and Atlas working well together. There are very minor new features, but they are of trivial significance, its mostly about confirming that existing functionality still works. CLI should be a bit more involved, as there will be some new non-trivial features. Part of the rationale for the testing is also to help get informed for updating tutorials.
Current Status
Nothing has been done.
Initial Division of labour
Not determined.
Community and Communications
Team
Martin
Ben
Work Scope
Update website for announcement.
Update website for release.
Update tutorials.
Announcement and release communications: blog + newsletter
Current Status
Nothing has been done.
Initial Division of labour
Not determined.
Questions
Is there a way for us to not introduce Hydra subscriptions to indicate having processed a block in order to allow apps to know when they can expect the query node to have incorporated the effect of some transaction they just issued (Transaction UX hydra#166)? We could just have fixed delay after confirmation of transaction from full node? How much worse is this?
Could we find a similar workaround to avoid needing subscriptions for metatransaction processing confirmation (Metaprotocol DevUX #2018)? The delay could be the same length, but there is no way to know whether the operation succeeded or not beyond making it into a block.
How are we currently doing filtering is search to avoid uploads that are too new? client-side filtering? if so, how well does this work for complex checks like we need now, how much does it distort the search result?
What would be the effect of trying to update the content directory integration tests for this release, while there are independent changes already done for Olympia? What are reconciliation costs?
Dependencies
Project Management
Release Manager
Bedeho
The release manager is responsible for tracking overall progress, detecting delays and course-correcting. Tracking will be done weekly in this issue
A release wide meeting will be organised semi-regularly, we could start at a pace of once ever 2 weeks to begin with. Experience has shown that, even when there is no obvious issues that need tackling, discussions often reveal issues and corner cases that have not been accounted for. We can aim for one team rep. being present, so as not to distract everyone. Log will be kept here:
This is a preliminary document, it must be checked by key contributors to give input on expected full costs of various intended changes, and also whether goals in one area are fully reflected in requirements of dependant areas.
Name
SumerBrand
Goals
1.
Deploy metaprotocol content directory.2.
Publishing in Atlas.Overview
AlertIn the end, make sure you understand this example to see how everything on the infrastructure+runtime side is brought together, even if this is not your primary focus area
#2032
Background
This release is a synthesis of distinct strands of improvement which became most practical to plan and execute in one release due to the interdependencies involved:
In what follows we will dive more deeply into each release goal, which then touches upon the
Metaprotocol content directory
The currently deployed content directory, described here
https://joystream.gitbook.io/joystream-handbook/subsystems/content-directory
has a few problems
the complexity comes from two sources
neither of these actually made much sense in the full context of what we are doing. For 1, higher levels of the stack, like the query node and apps, end up having too validation to generate convenient statically type safe interfaces anyway, and for 2 it is just too complex yet does not allow us to do new things we come up with, so its a bad tradeoff. Therefore we made a smart contract version of the content directory where metadata was not validated by the contract, this solves 1 and 2. The approach taken is described here
This solution was much better, but it became clear that even smart contracts that did not look at metadata were really a suboptimal half way house, and that a so called metaprotocol would be the best tradeoff available, see full description here
The result has therefore been that we will
Publishing in Atlas
Publishing Today
Current it is not possible to publish in atlas, in fact the only way to create and manage channels and videos is using a CLI, which is extremely cumbersome, as it requires learning about and correctly doing the following
Publishing in Atlas
A full onboarding, membership creation, channel creation, dashboard, video publishing and management experience has been designed, see here
https://www.figma.com/file/rBjfm2tJ2Pr9M0UNTduU9v/Joystream-Atlas?node-id=3079%3A30013
Importantly, a broader set of features have been designed than we should aspire to build in this release, so we are focusing on a core subset of features related to publishing, without accumulating tech debt. It was originally done under the following design brief,
Joystream/atlas#31
which itself is not in synch with the final designed product, because we decided to change some of the constraints in process - most importantly having Joystream hosted media assets.
However, mistakes were made on the part of Bedeho when directing this effort, hence some mismatches have been identified between what the design presumes and the behaviour of the system, and some other more minor areas of improvement, see here
Joystream/atlas#215
Work has started to analyse the required changes, and the final results should be reflected in the final release plan.
Runtime
Team
Work Scope
Remember no benchmarking and fees as we are basing ourselves on the Babylon runtime.
Initial Division of labour
Shamil does everything.
Content Directory
Team
Work Scope
Current Status
Nothing is done here.
Initial Division of labour
Mokhtar only.
Hydra
Team
Work Scope
Current Status
Nothing is done here.
Initial Division of labour
Metin only.
Query Node
Team
Work Scope
Current Status
Nothing is done here.
Initial Division of labour
Atlas
Team
Work Scope
Initial Division of labour
Klaudiusz will allocate tasks internally in Atlas team.
CLI & Tooling
Team
Work Scope
Current Status
Nothing is done here.
Initial Division of labour
Unclear, perhaps Mokhtar focuses on CLI and Arsen on the mass upload script?
Network Testing
Team
Work Scope
Current Status
Nothing has been done.
Initial Division of labour
Mokhtar does everything.
Deployment
Team
Work Scope
Current Status
Nothing has been done.
Initial Division of labour
Unclear.
Manual QA
Team
Work Scope
Current Status
Nothing has been done.
Initial Division of labour
Not determined.
Community and Communications
Team
Work Scope
Current Status
Nothing has been done.
Initial Division of labour
Not determined.
Questions
Dependencies
Project Management
Release Manager
BedehoThe release manager is responsible for tracking overall progress, detecting delays and course-correcting. Tracking will be done weekly in this issue
#2002
as described by the following methodology
#474
Release Branch
https://github.com/Joystream/joystream/tree/sumer
Github Project(s)
We are doing everything in one project.
https://github.com/orgs/Joystream/projects/35
Team Meetings
Biweekly(ish) Release Meetings
A release wide meeting will be organised semi-regularly, we could start at a pace of once ever 2 weeks to begin with. Experience has shown that, even when there is no obvious issues that need tackling, discussions often reveal issues and corner cases that have not been accounted for. We can aim for one team rep. being present, so as not to distract everyone. Log will be kept here:
#1241
The text was updated successfully, but these errors were encountered: