Example Workflow: XBrain

This example, running via Airflow on AWS batch, pulls data from the Block Object Storage Service (BOSS), runs data through the xbrain workflow and uploads the data back to the BOSS, utilizing S3 as an intermediary storage location.

Building tool containers

Before running this workflow, its important to build the tool containers. This can be done by navigating to the appropriate tools and running docker build.

For this example, two containers need to be built -- xbrain and boss-access.

1. Navigate to the xbrain dockerfile location

   cd saber_private/saber/xbrain/
   

   And build and tag the container. Notice we give it a tag of aplbrain/xbrain. This particular tag can be whatever you want, but make sure to change it in the tool definition (found in saber_private/saber/xbrain/tools)

   docker build -t aplbrain/xbrain .

2. Now navigate to the boss_access dockerfile location and build that container. Note that the tool cwl files are stored in the same directory as the dockerfile. This is also allowed, as long as the workflow cwl references them properly.

   cd ../boss_access
   docker build -t aplbrain/boss-access .
   

Now that the tool containers are built, you can move onto the next step.

Setting parameters

Parameters are set in a yaml file, hereby referred to as a job file. For this example, this can be found in saber/xbrain/example/example-job.yml. If you open this file, you will see the many parameters of the workflow. It shouldn't be necessary to change anything here to run the demo.

Compiling and launching the workflow

1. Open a bash shell in the cwl_parser container (you can find the name on your system, after running docker-compose up, using docker container list, looking for the image with the name CWL_parser)

   docker exec -it conduit_docker_cwl_parser_1 /bin/bash
   

2. Navigate to the xbrain job folder and run the conduit build command.

   cd saber/xbrain/example/
   conduit build workflow.cwl example-job.yml
   

   This parses the CWL workflow and the job definition, creates the job definitions on AWS batch, and pushes the tool containers.

3. Now create the Airflow DAG with conduit parse.

   conduit parse workflow.cwl example-job.yml
   

   This registers the workflow with Airflow.

4. Now the workflow is ready to run. Go to the web interface and click the on play button near the dag.