From e1eb556ccfa70691d449b5a1c88ef1046e7f0414 Mon Sep 17 00:00:00 2001 From: Tim Paine <3105306+timkpaine@users.noreply.github.com> Date: Fri, 16 Aug 2024 19:53:55 -0400 Subject: [PATCH] update docs --- README.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 26fd053..2f82ba0 100755 --- a/README.md +++ b/README.md @@ -13,8 +13,27 @@ This library provides a configuration-driven way of generating [supervisor](http ## How To: Use in Airflow -Docs coming soon! +`airflow-supervisor` can be installed in your airflow server environment and imported in your dag files. It provides two convenient top level DAG subclasses: -## How To: Use as a standalone supervisord frontend +- `SupervisorLocal`: creates a DAG representing a local supervisor instance running on the airflow worker node (underlying task will use `PythonOperator` and `BashOperator` to communicate between airflow and supervisor) +- `SupervisorRemote`: creates a DAG representing a remote supervisor instance running on another machine (underlying tasks will use `SSHOperator` to communicate between airflow and supervisor) + +We expose DAGs composed of a variety of tasks and sensors, which are exposed as a discrete pipeline of steps: +1. Setup `supervisord` configuration +2. Start the `supervisord` daemon +3. Start the supervised programs with `supervisorctl` +4. Start sensors to query the programs' state via [supervisor's XML-RPC API](http://supervisord.org/api.html) +5. Evaluate and take action according to the program's state changes +6. Restart programs if necessary +7. Tear down the sensors from (4) +8. Stop the supervised programs from (3) +9. Stop the `supervisord` daemon from (2) +10. Remove configuration from (1) + +This setup provides maximal configureability with a minimal requirements on the machine (for example, no requirements on an existing `supervisord` daemon via e.g. `systemd`). It also lets you hook your own tasks into any step of the process. For example, if we detect a process has died in step (5), you could configure your own task to take some custom action before/instead of the default restart of step 6. + +More docs and code examples coming soon! + +## How To: Use as a supervisord configuration frontend This library can be used outside airflow as a generic supervisord configuration framework, with the static typing benefits that entails. For an example, look at the [hydra configuration test](./airflow_supervisor/tests/hydra/test_hydra.py). This example generates a supervisor configuration file by composing independent hydra configs.