Introduce public application/extension API for Superdesk and its apps

[tomaskikutis]

I'll start with a problem I want this to solve: Superdesk is a lot harder/slower to develop and buggier than it should be. I believe the main cause of this is the lack of strict boundaries between features. Features depend on internal implementation details of other features and the core. This makes both the core and features harder to change and more prone to bugs. On top of that, every time we beat the difficulties and do the change, it gets even more difficult next time, because we had added even more inter-dependencies in the previous hard change.

I suggest introducing the boundaries in the form of a public application/extension API. I'm not yet sure how to call it, so I'll just call it "the API" here. The point is that the core should not know anything about the apps, and the apps should not know anything about the core or other apps, except what all expose through their public APIs. I would like the API to be as stateless, immutable, side-effect free and low-knowledge as possible. By low-knowledge I mean that we should hide as much implementation details as possible and not expose things if we can implement what we need without exposing it.

By having such API, we could move a lot of features out of our long overgrown core and still have multiple client repositories configured to use these features.

For example, editor3 supports comments and requires that articles aren't published until the comments are resolved. Instead of the core knowing about the comments feature directly, it should instead allow apps to register middlewares in which case an app could then cancel publishing itself via the API.

Another good example is a feature Petr was working on where it was required to append text to a headline if certain genre/priority was selected. The maximum length of the headline had to be adjusted according to the text appended. This is a very specific requirement which shouldn't be hardcoded into the core, but couldn't have been implemented otherwise without the API.

My examples are only from client-core, but we should apply the same principles on the back-end as well.

Here are some related discussions we already had:

• https://sourcefabric.slack.com/archives/C1XBKE70C/p1554989580013500
• feat(authoring): add limits service superdesk-client-core#2929 (comment)
• [SDESK-4128] FunctionPoint service for generic function extensions superdesk-client-core#2930

I've implemented the management of annotations(SDFID-525) and applying them in an editor(SDFID-524) in a client repo, which turn out needs to be available to other clients. That looks like a good opportunity to start building the API. The features are unrelated from responsibilities of a core, so I'd like to create a repository for them. Something like superdesk-modules or superdesk-extensions. The repo would contain a folder for the app and its client and server subfolders, similarly how it's done in planning or analytics modules. We could create a separate repo per app, but I think it would be easier to manage if apps were in one repo.

[mdhaman]

If we have multiple repo per app then releasing changes to production would be problematic, it would be easier with single repo.

[tomaskikutis]

How would it make releasing problematic? On the other hand, I don't mind keeping extensions in the core repo if they can't access any of the core, but the API.

[mdhaman]

You can have superdesk-modules as separate single repo but having separate repos per app would cause issue with releasing as you would need to keep track of the dependencies. For example, if a developer uses a functionality from core (master branch) in planning or analytics module but in production you are using 1.28 of core then it is not guaranteed to work with version 1.28 as all our testing is done against master branch. Same applies if you use a css class or component from UI framework and your production version depends on older version of UI framework. Hence if you have more repos then keeping track of dependencies become harder.

[MarkLark86]

What if we were to use lerna (https://github.com/lerna/lerna) to manage each monorepo. Each repo in github would be a module and have a folder structure like the following:

• core
  • package.json
  • packages:
    • core-factory
    • core-db
    • core-api
    • core-utils

And each package in the module could have the following folder structure:

• core-factory:
  • client
    • package.json
    • core-factory
      • ....code
  • server
    • core-factory
    • ....code

We could have a parent module that depends on the packages such as:

• "@superdesk/core-factory": "1.30.x"
• "@superdesk/core-db": "1.30.x"
• "@superdesk/core-api": "1.30.x"
• "@superdesk/core-utils": "1.30.x"

Then register the module to npm under a user scoped name such as @superdesk/core (https://docs.npmjs.com/about-scopes). The packages themselves could also be registered to npm if required, such as @superdesk/analytics-utils.

The dependencies in package.json could then require modules or packages:

• "@supredesk/core": "1.30.x"
• "@superdesk/ui-framework": "1.18.x"
• "@superdesk/planning": "1.5.x"
• "@superdesk/analytics-utils": "1.7.x"

This setup could gradually be introduced as each module is being developed with minimal changes to current package.json files.

Thoughts?

[MarkLark86]

We would also require something like lerna for the python side, and register those modules/packages with pip as well.

We could then manage these modules/entire system with a superdesk cli (something I created a while ago - https://github.com/marklark/pyman, or any other library that meets the requirements). We would then be able to automate the above functionality, and provide an interface to run any of the flask commands (each grouped under a category in the cli). This would make it easier to run a command that you haven't run in a long time, or for newcomers learning to setup their environment.

[tomaskikutis]

You can have superdesk-modules as separate single repo but having separate repos per app would cause issue with releasing as you would need to keep track of the dependencies.

I totally agree. I didn't suggest creating a repo per extension though. My original proposal was one separate repo for all extensions. When I think more about it, extensions folder in the core might even be easier to maintain. API methods are likely to be added frequently in the beginning and we would save some time by not having to manage the same PR across multiple repositories.

What if we were to use lerna (https://github.com/lerna/lerna) to manage each monorepo.

Having a dedicated repository per extension is unarguably harder to manage. Tools like lerna would definitely help, but I don't see any benefit we would get by having a repo per extension. We can have separate dependencies & separate package.json files even in one repository similarly how vscode does. The same could work on the server side with a separate requirements file for each extension.

[MarkLark86]

I was thinking of the following modules:

• core (core functionality for a web app)
• authoring (the Superdesk app)
• planning
• analytics

Separating out the core and Superdesk app would make it easier to manage the framework vs application layers, as well as use the core module for another web app, i.e. Newsroom/Newshub.

This way version 2 of the core could be worked on to support Elasticsearch 6, then once that's completed work can begin on version 2 of authoring to use the newer elastic

[tomaskikutis]

What is hard about managing framework vs application layers, and how a separate repository would make this easier?

You mentioned an example with different apps using different versions of dependencies(elastic in your example), but isn't this a responsibility of a build process? Can't you adjust build processes to enable 2 apps to use different versions of dependencies even in a single repository?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tomaskikutis]

Implemented in superdesk/superdesk-client-core#2950 🎉