Compas framework

[dirkdev98]

Tracking issue to make a Compas framework. For the why and how, check out #2774 (comment)

• Output modes:
  • All output systems are disabled by default, and should be enabled manually based on the actions done.
  • TUI: A UI for development. Should show active information, available actions and able to attach any stream
  • Debug: Write debug information to a file to get better insights in what's happening under the hood
  • Log: A normal logger to use in any other command
• Configuration loading
  • Workspaces support
  • Cast validation errors to human readable errors
• Detect different modes like development, production and CI
  • This allows us to ingrate all further tasks based on
• Caching and file watching
  • Cache the resolved configs
  • Only resolve configs again if the file watcher notifies on changes to config files
  • Full cache should automatically bust if a different Compas version is used
• Configurable TUI actions
  • Setup TUI actions in the configs
  • Resolve an stack of action views and their inputs
  • Navigate through the actions
  • Dynamic process manager + process manager actions like restarting, stopping, etc
  • This should make the zakmes CLI kind-of usable
• Auto-installation
  • If development and no CI, auto-install compas as dev dependency
  • If npx -p compas zakmes init auto-install as prod dependency
  • Setup executable shell script (which integrates with package manager below)
• Package manager integrations
  • Determine and cache the package manager that is used
  • Watch on package.json changes
  • Verify if top-level package versions are up-to-date
  • Propose to run $pkg install
  • Requires dynamic actions in the TUI
• Overtake compas docker commands
  • This would be the start of the zakmes init docker and related commands. We want people to use the zakmes CLI in a variety of scenario's. So we shouldn't automatically boot up Docker in any scenario, like for example a frontend project.
  • Automatically boot up docker containers
  • Allow configuring containers in the config
• Overtake compas lint
  • Expose zakmes init lint
  • Integrate with file watcher to detect last lint run and propose linting of specific files
  • Shouldn't be done automatically in the dev server since that could modify the file the user is currently working in.
• Integrate running migrations in the dev server
  • Add zakmes init migrations to add @compas/store & migrations/.gitkeep
  • Automatically enable when @compas/store is installed and a migrations dir exists
  • Watch .sql & .js files in the migrations directories.
  • Acquire migration info and propose a migration run / full clean & retry
• Work out zakmes ci
  • zakmes init github should add some default workflow files
  • Template with inferred package manager, install dependencies
  • Start docker containers, run migrations
  • Run inferred Prettier & ESLint
  • Run inferred test command
• Define and implement how we are going to auto-load queue handlers
  • How to add global options?
  • Test primitives?
  • Should be statically analysable
  • Automatically mutate with JSDoc blocks (or should this be a dynamic action)
  • Requires more framework scaffolding like Postgres connection management
• Work out zakmes prod & zakmes prod queue
  • What happens if multiple queue's are defined?
  • Should scripts/queue.js override behaviour?
• Integrate with create-compas
  • Example for a setup with api and queue
  • Example for a setup for library development
  • Example for a setup with a frontend framework like Next.js
  • Example for a setup with workspaces
• TBD:
  • zakmes verify load everything, without executing anything
  • Include compas migrate
  • Include compas run ...
  • Include compas test
  • Include zakmes prod
  • Include code-gen
  • More frameworky things like auto-loading controller files etc.

[dirkdev98]

Why

The time has come, we are going full steam ahead in to framework mode. Starting a new Compas project from scratch currently involves a lot of boilerplate. Looking at the with-auth template, 7 folders and many more files just to run code generation, an api, a queue worker and of course tests. Most of it is just wiring things up in one of the few ways possible. We can do that automatically as well.

What

What if you could just run compas in your terminal and get all manual labor done automatically. It will do the correct thing for libraries, backends and even in frontend projects. Managing the required docker containers. Automatically generating, without busting lint and build caches. Keeping track of which order of tests fail when running compas test --randomize-rounds 3 so you can debug that flaky situation.

How

Not every backend is the same of course. Some require a different Postgres version. Others connect to external api's and would prefer to use code-gen compatible with OpenAPI specs. And we understand that you don't want to rewrite your existing backends to folder based routing. This means that we are slowly going move things in to the compas package and keep a lot of things configurable with sensible defaults.

The first change in this direction has already been started. We are going to purpose the compas package for the new CLI. It will contain a new (temporary) CLI called zakmes, which will directly showcase the new terminal user interface (TUI) of the dev server. We will slowly expand its functionality, either by using the existing CLI or taking over it's functionality.

A peek ahead

We are aiming for the following behaviour...

... in development

> compas

Loaded config...
Generating...
Launching prerequisites...
Starting watcher...
Waiting on syntax error in `src/foo/bar.js`...
Files are waiting to be linted...

# Input: L

Running the linter for 5 files...
All files are up-to-date.

... in CI

> compas

Loaded config...
Generating...
Launching prerequisites...
Running linter...
Running tets...

Done!

... in production

> compas prod

Loaded config...
Generating...
Starting api...
Starting queue...

# Or

> compas prod api

Loaded config...
Generating...
Starting api...

Conclusion

The road up ahead is exciting! There will be some bumps and breaking changes for sure. But the result should be faster, better integrated and most importantly more stable to build on.