Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
timkpaine committed Aug 16, 2024
1 parent 68f083f commit e1eb556
Showing 1 changed file with 21 additions and 2 deletions.
23 changes: 21 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,27 @@ This library provides a configuration-driven way of generating [supervisor](http

## How To: Use in Airflow

Docs coming soon!
`airflow-supervisor` can be installed in your airflow server environment and imported in your dag files. It provides two convenient top level DAG subclasses:

## How To: Use as a standalone supervisord frontend
- `SupervisorLocal`: creates a DAG representing a local supervisor instance running on the airflow worker node (underlying task will use `PythonOperator` and `BashOperator` to communicate between airflow and supervisor)
- `SupervisorRemote`: creates a DAG representing a remote supervisor instance running on another machine (underlying tasks will use `SSHOperator` to communicate between airflow and supervisor)

We expose DAGs composed of a variety of tasks and sensors, which are exposed as a discrete pipeline of steps:
1. Setup `supervisord` configuration
2. Start the `supervisord` daemon
3. Start the supervised programs with `supervisorctl`
4. Start sensors to query the programs' state via [supervisor's XML-RPC API](http://supervisord.org/api.html)
5. Evaluate and take action according to the program's state changes
6. Restart programs if necessary
7. Tear down the sensors from (4)
8. Stop the supervised programs from (3)
9. Stop the `supervisord` daemon from (2)
10. Remove configuration from (1)

This setup provides maximal configureability with a minimal requirements on the machine (for example, no requirements on an existing `supervisord` daemon via e.g. `systemd`). It also lets you hook your own tasks into any step of the process. For example, if we detect a process has died in step (5), you could configure your own task to take some custom action before/instead of the default restart of step 6.

More docs and code examples coming soon!

## How To: Use as a supervisord configuration frontend

This library can be used outside airflow as a generic supervisord configuration framework, with the static typing benefits that entails. For an example, look at the [hydra configuration test](./airflow_supervisor/tests/hydra/test_hydra.py). This example generates a supervisor configuration file by composing independent hydra configs.

0 comments on commit e1eb556

Please sign in to comment.