feat(cli): Experimental support for plugins

[Josh-Walker-GM]

Warning

This is an experimental feature.

Problem

We want Redwood's CLI to be more pluggable. It should start to support plugins so we as a framework can write smaller individual CLI plugin packages which should be more performant and maintainable. We also need to support third party plugins so our partners and community can develop their own plugins for the redwood CLI.

Note

I have used the term "plugin" which may not follow the conventional nomenclature for this sort of feature as it currently exists. This feature at the moment is a mechanism to provide third party (non-redwoodjs) commands to the redwood CLI. It is not a mechanism for "hooking into" or otherwise injecting functionality into existing redwood processes.

Solution

Note

This PR does not split up the CLI into any packages and does not introduce any breaking changes. It merely adds the functionality to load plugins should they be present which by default they are not.

We support arbitrary npm packages that export a field commands which is a list of yargs command objects. These plugin packages are defined in the redwood.toml file with the following structure (the example is a valid third party package if you wish to test):

[experimental.cli]
  autoInstall = false
  [[experimental.cli.plugins]]
    package = "@joshgmw/rw-plugin-one"
    version = "0.0.3"

Packages must have a package and a version and can optionally contain an enabled flag which defaults to true. The package is the npm package name and version is the npm package version, semver or tag.

autoInstall is a flag which controls whether the CLI should attempt to automatically install any plugins which are defined in the TOML but which are not currently installed as a dev dependency. It currently defaults to false.

Plugin packages must be published under a scope such that they obtain an @author/ prefix to the package name. This is because plugin commands are "namespaced" under the npm scope they were published under. Such that attempting to call commands from the example third party plugin above is achieved by:

yarn rw @joshgmw --help

This namespacing is nessecary to avoid global command name collisions. Without namespacing two plugins which have an setup command would conflict and yargs would only dispatch to the first setup which it loaded. With namespacing the task of preventing collisions belongs to plugin authors and they only need to avoid conflicts with any other plugins published under their scope.
RedwoodJS has reserved the '' namespace so that all plugins published under @redwoodjs/ don't need a namespace prefix, retaining their usual form of yarn rw [command]

Optimisations

Although the goal is to support plugins we must be careful to ensure the CLI is performant - both with disk usage and process memory. A few simple optimisations have been added here:

• Namespace-based plugin culling

  This means any plugin which are not from the namespace included in the invoked command are not loaded into memory. E.g. we only load plugins from @joshgmw/ when we run yarn rw @joshgmw [command] as we know @jtoar plugins will not be needed.

• Command-based plugin culling

  It would still be non-performant to load all plugins within the same namespace. @redwoodjs/ will likely publish multiple plugins and we do not wish to load all of them into memory to support a command which will only be performed by one of those packages.

  To avoid this we currently cache the top level commands a plugin provides and use this cache to opportunistically skip loading them if we are sure they cannot provide the command we wish to call. This caching currently requires the plugin to have been loaded into memory at least once before so the cache entry is present.

  Example: Assume @joshgmw/rw-1 provides generator commands under yarn rw @joshgmw generate [type] and that @joshgmw/rw-2 provides setup commands under yarn rw @joshgmw setup [type]. Now assume a user runs yarn rw @joshgmw setup coolThing. By sampling the first command word, setup, and using the cache entries we can eliminate the need to load @joshgmw/rw-1 because we know it cannot satisfy a setup command.

Those optimisations should ensure only the dependencies of the packages which could provide the command are loaded into memory. Smaller CLI plugin packages which contain the minimal dependencies will therefore reduce memory usage.
To reduce initial disk size we can make use of lazy-installing plugin packages. Plugins are installed as dev dependencies and plugins are known to the CLI via the TOML. This means it's possible for the CLI to know about the existence of commands which have not yet been fully installed. By selectively including only a subset of command packages as dev dependencies at the time we install a users project we can reduce the initial disk usage. The CLI plugin packages will be installed when needed.

Note

As the cache entry for a plugin is currently only available after it is loaded for the first time we will only be able to lazy-install at the namespace level currently rather than at the individual plugin level. Providing a precomputed cache would allow full lazy-installing capabilities.

Next steps (in following PRs)

1. We should start splitting up the existing CLI package into smaller packages based on:
   i. Usage volume - niche commands can be in a package that isn't installed in a users project at install time
   ii. Scope/theme - commands with similar actions should be grouped into the same package
   iii. Need - Commands that are immediately essential at dev time, like dev, build or serve should be grouped into a package
2. Add a CI action to precompute the cache for @redwoodjs/ cli plugins when they change and add that to the CRWA template. This would be an initial stopgap solution to an online cache entry.
3. Add helper functions to the base CLI package to assist in updating the cli.plugins TOML entries. Commands like add, remove, and update which assist in maintaining the plugin entries.
4. Support some sort of online registry for users to see what packages exist and to provide an API to access data such as the cache entry.

Note: The instrumentation of the CLI using OpenTelemetry is likely to start alongside the process of introducing this plugin based architecture.

[replay-io]

18 replays were recorded for 80a7d4c.

0 Failed

18 Passed

• useAuth hook, auth redirects checks
• Check that a specific blog post is prerendered
• Check that about is prerendered
• Check that homepage is prerendered
• Check that meta-tags are rendering the correct dynamic data
• Check that rehydration works for page not wrapped in Set
• Check that rehydration works for page with Cell in Set
• Check that rehydration works for page with code split chunks
• Check that you can navigate from home page to specific blog post
• Waterfall prerendering (nested cells)
• RBAC: Admin user should be able to delete contacts
• RBAC: Should not be able to delete contact as non-admin user
• Smoke test with dev server
• Smoke test with rw serve
• Loads Cell mocks when Cell is nested in another story
• Loads Cell Stories
• Loads MDX Stories
• Mocks current user, and updates UI while dev server is running

View test run on Replay ↗︎

[thedavidprice]

looping in @orta for visibility

[jtoar]

Awesome stuff @Josh-Walker-GM! I ran some preliminary benchmarks and I can't really argue with them. Right now the Redwood commands are virtually the same and the plugin example you provided for testing (thank you 🙏) is much faster, which bodes well.

I just did a preliminary review; most of my comments are nitpicks, but the theme of it is "can we make the code a little more readable"? I'm sure some of it is that it's just a new concept I have to get familiar with. (I'll do another round today, I want to understand the for loop on line 105 a bit more. But figured I'd let you wallow in the nits for now. Sorry. I'll add that none of my comments are blockers)

Last nit 😅 we should probably go with [experimental.cli] because, well 1) it's experimental 2) we had confusion around that fact already, and 3) I bet if a user saw this table in the TOML right now they'd just assume it's stable.

And re next steps:

We should start splitting up the existing CLI package into smaller packages based on...

This is what I'm most excited for!

[jtoar]

@Josh-Walker-GM could you help me understand when this condition would be met?

[Josh-Walker-GM]

Right now this occurs when we've filtered out all possible commands. This might happen for example if we could tell from the cache that no plugins could possibly satisfy the command you entered.

This raises the point that the help message is not great in that scenario I give above. We didn't load the plugins into memory because we know none can satisfy the user but as a result we don't get a very good help message. For example if I run yarn rw @joshgmw testing - which no @joshgmw plugin provides then I get the following as the last line of the output:

Unknown arguments: @joshgmw, testing

and the default yargs message only contains any other plugins that are registered. Right now that includes all the built-in CLI commands but in the future there will be none or only a few basic ones.

Maybe we should in that case go back and load them all into memory and yargs will then produce the appropriate help message. This would take a little refactor to loop round again and load all the plugins. In the meantime I have added a console warning pointing the user to run yarn rw @namespace --help

[Josh-Walker-GM]

I'll refactor this later tonight when I get some more time so that it just loads all the plugins and provides the standard yargs help.

[jtoar]

Looks great! At this point most of the comments I was about to write don't feel helpful for the stage we're at. Thanks for taking my feedback in stride—let's keep going

[notion-workspace]

Breakup current CLI into plugins