diff --git a/.goreleaser.yaml b/.goreleaser.yaml index 47be34630..bdf2efea6 100644 --- a/.goreleaser.yaml +++ b/.goreleaser.yaml @@ -34,7 +34,6 @@ brews: - repository: owner: yohamta name: homebrew-tap - folder: Formula homepage: "https://github.com/dagu-dev/dagu" description: "A No-code workflow executor that runs DAGs defined in a simple YAML format" license: "GNU General Public License v3.0" @@ -48,8 +47,7 @@ brews: end - repository: owner: dagu-dev - name: homebrew-dagu - folder: Formula + name: homebrew-brew homepage: "https://github.com/dagu-dev/dagu" description: "A No-code workflow executor that runs DAGs defined in a simple YAML format" license: "GNU General Public License v3.0" diff --git a/README.md b/README.md index e76f8aa73..458be0f3e 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@
-[Installation](https://dagu.readthedocs.io/en/latest/installation.html) | [Community](https://discord.gg/gpahPUjGRk) | [Quick Start](https://dagu.readthedocs.io/en/latest/quickstart.html) +[Installation](https://dagu.readthedocs.io/en/latest/installation.html) | [Community](https://discord.gg/gpahPUjGRk) | [Quick Start](https://dagu.readthedocs.io/en/latest/quickstart.html)
@@ -28,85 +28,86 @@ Dagu is a powerful Cron alternative that comes with a Web UI. It allows you to define dependencies between commands as a [Directed Acyclic Graph (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph) in a declarative [YAML format](https://dagu.readthedocs.io/en/latest/yaml_format.html). Dagu simplifies the management and execution of complex workflows. It natively supports running Docker containers, making HTTP requests, and executing commands over SSH. -- [Documentation](https://dagu.readthedocs.io) -- [Localized Documentation](#localized-documentation) -- [Discord Community](https://discord.gg/gpahPUjGRk) +- [Documentation](https://dagu.readthedocs.io) +- [Localized Documentation](#localized-documentation) +- [Discord Community](https://discord.gg/gpahPUjGRk) ## **Highlights** -- Single binary file installation -- Declarative YAML format for defining DAGs -- Web UI for visually managing, rerunning, and monitoring pipelines -- Use existing programs without any modification -- Self-contained, with no need for a DBMS + +- Single binary file installation +- Declarative YAML format for defining DAGs +- Web UI for visually managing, rerunning, and monitoring pipelines +- Use existing programs without any modification +- Self-contained, with no need for a DBMS ## **Table of Contents** -- [**Highlights**](#highlights) -- [**Table of Contents**](#table-of-contents) -- [**Features**](#features) -- [**Use Cases**](#use-cases) -- [**Web UI**](#web-ui) - - [DAG Details](#dag-details) - - [DAGs](#dags) - - [Search](#search) - - [Execution History](#execution-history) - - [Log Viewer](#log-viewer) -- [**Installation**](#installation) - - [Via Bash script](#via-bash-script) - - [Via GitHub Releases Page](#via-github-releases-page) - - [Via Homebrew (macOS)](#via-homebrew-macos) - - [Via Docker](#via-docker) -- [**Quick Start Guide**](#quick-start-guide) - - [1. Launch the Web UI](#1-launch-the-web-ui) - - [2. Create a New DAG](#2-create-a-new-dag) - - [3. Edit the DAG](#3-edit-the-dag) - - [4. Execute the DAG](#4-execute-the-dag) -- [**CLI**](#cli) -- [**Localized Documentation**](#localized-documentation) -- [**Documentation**](#documentation) -- [**Running as a daemon**](#running-as-a-daemon) -- [**Example DAG**](#example-dag) -- [**Motivation**](#motivation) -- [**Why Not Use an Existing DAG Scheduler Like Airflow?**](#why-not-use-an-existing-dag-scheduler-like-airflow) -- [**How It Works**](#how-it-works) -- [**License**](#license) -- [**Support and Community**](#support-and-community) +- [**Highlights**](#highlights) +- [**Table of Contents**](#table-of-contents) +- [**Features**](#features) +- [**Use Cases**](#use-cases) +- [**Web UI**](#web-ui) + - [DAG Details](#dag-details) + - [DAGs](#dags) + - [Search](#search) + - [Execution History](#execution-history) + - [Log Viewer](#log-viewer) +- [**Installation**](#installation) + - [Via Bash script](#via-bash-script) + - [Via GitHub Releases Page](#via-github-releases-page) + - [Via Homebrew (macOS)](#via-homebrew-macos) + - [Via Docker](#via-docker) +- [**Quick Start Guide**](#quick-start-guide) + - [1. Launch the Web UI](#1-launch-the-web-ui) + - [2. Create a New DAG](#2-create-a-new-dag) + - [3. Edit the DAG](#3-edit-the-dag) + - [4. Execute the DAG](#4-execute-the-dag) +- [**CLI**](#cli) +- [**Localized Documentation**](#localized-documentation) +- [**Documentation**](#documentation) +- [**Running as a daemon**](#running-as-a-daemon) +- [**Example DAG**](#example-dag) +- [**Motivation**](#motivation) +- [**Why Not Use an Existing DAG Scheduler Like Airflow?**](#why-not-use-an-existing-dag-scheduler-like-airflow) +- [**How It Works**](#how-it-works) +- [**License**](#license) +- [**Support and Community**](#support-and-community) ## **Features** -- Web User Interface -- Command Line Interface (CLI) with several commands for running and managing DAGs -- YAML format for defining DAGs, with support for various features including: - - Execution of custom code snippets - - Parameters - - Command substitution - - Conditional logic - - Redirection of stdout and stderr - - Lifecycle hooks - - Repeating task - - Automatic retry -- Executors for running different types of tasks: - - Running arbitrary Docker containers - - Making HTTP requests - - Sending emails - - Running jq command - - Executing remote commands via SSH -- Email notification -- Scheduling with Cron expressions -- REST API Interface -- Basic Authentication over HTTPS +- Web User Interface +- Command Line Interface (CLI) with several commands for running and managing DAGs +- YAML format for defining DAGs, with support for various features including: + - Execution of custom code snippets + - Parameters + - Command substitution + - Conditional logic + - Redirection of stdout and stderr + - Lifecycle hooks + - Repeating task + - Automatic retry +- Executors for running different types of tasks: + - Running arbitrary Docker containers + - Making HTTP requests + - Sending emails + - Running jq command + - Executing remote commands via SSH +- Email notification +- Scheduling with Cron expressions +- REST API Interface +- Basic Authentication over HTTPS ## **Use Cases** -- **Data Pipeline Automation:** Schedule ETL tasks for data processing and centralization. -- **Infrastructure Monitoring:** Periodically check infrastructure components with HTTP requests or SSH commands. -- **Automated Reporting:** Generate and send periodic reports via email. -- **Batch Processing:** Schedule batch jobs for tasks like data cleansing or model training. -- **Task Dependency Management:** Manage complex workflows with interdependent tasks. -- **Microservices Orchestration:** Define and manage dependencies between microservices. -- **CI/CD Integration:** Automate code deployment, testing, and environment updates. -- **Alerting System:** Create notifications based on specific triggers or conditions. -- **Custom Task Automation:** Define and schedule custom tasks using code snippets. +- **Data Pipeline Automation:** Schedule ETL tasks for data processing and centralization. +- **Infrastructure Monitoring:** Periodically check infrastructure components with HTTP requests or SSH commands. +- **Automated Reporting:** Generate and send periodic reports via email. +- **Batch Processing:** Schedule batch jobs for tasks like data cleansing or model training. +- **Task Dependency Management:** Manage complex workflows with interdependent tasks. +- **Microservices Orchestration:** Define and manage dependencies between microservices. +- **CI/CD Integration:** Automate code deployment, testing, and environment updates. +- **Alerting System:** Create notifications based on specific triggers or conditions. +- **Custom Task Automation:** Define and schedule custom tasks using code snippets. ## **Web UI** @@ -114,34 +115,34 @@ Dagu is a powerful Cron alternative that comes with a Web UI. It allows you to d It shows the real-time status, logs, and DAG configurations. You can edit DAG configurations on a browser. - ![example](assets/images/demo.gif?raw=true) +![example](assets/images/demo.gif?raw=true) - You can switch to the vertical graph with the button on the top right corner. +You can switch to the vertical graph with the button on the top right corner. - ![Details-TD](assets/images/ui-details2.webp?raw=true) +![Details-TD](assets/images/ui-details2.webp?raw=true) ### DAGs It shows all DAGs and the real-time status. - ![DAGs](assets/images/ui-dags.webp?raw=true) +![DAGs](assets/images/ui-dags.webp?raw=true) ### Search It greps given text across all DAG definitions. - ![History](assets/images/ui-search.webp?raw=true) +![History](assets/images/ui-search.webp?raw=true) ### Execution History It shows past execution results and logs. - ![History](assets/images/ui-history.webp?raw=true) +![History](assets/images/ui-history.webp?raw=true) ### Log Viewer It shows the detail log and standard output of each execution and step. - ![DAG Log](assets/images/ui-logoutput.webp?raw=true) +![DAG Log](assets/images/ui-logoutput.webp?raw=true) ## **Installation** @@ -157,15 +158,16 @@ curl -L https://raw.githubusercontent.com/yohamta/dagu/main/scripts/installer.sh Download the latest binary from the [Releases page](https://github.com/dagu-dev/dagu/releases) and place it in your `$PATH` (e.g. `/usr/local/bin`). - ### Via Homebrew (macOS) + ```sh -brew install dagu-dev/dagu/dagu +brew install dagu-dev/brew/dagu ``` Upgrade to the latest version: + ```sh -brew upgrade dagu-dev/dagu/dagu +brew upgrade dagu-dev/brew/dagu ``` ### Via Docker @@ -190,22 +192,23 @@ Start the server and scheduler with the command `dagu start-all` and browse to ` Navigate to the DAG List page by clicking the menu in the left panel of the Web UI. Then create a DAG by clicking the `NEW` button at the top of the page. Enter `example` in the dialog. -*Note: DAG (YAML) files will be placed in `~/.dagu/dags` by default. See [Configuration Options](https://dagu.readthedocs.io/en/latest/config.html) for more details.* +_Note: DAG (YAML) files will be placed in `~/.dagu/dags` by default. See [Configuration Options](https://dagu.readthedocs.io/en/latest/config.html) for more details._ ### 3. Edit the DAG Go to the `SPEC` Tab and hit the `Edit` button. Copy & Paste the following example and click the `Save` button. Example: + ```yaml schedule: "* * * * *" # Run the DAG every minute steps: - - name: s1 - command: echo Hello Dagu - - name: s2 - command: echo done! - depends: - - s1 + - name: s1 + command: echo Hello Dagu + - name: s2 + command: echo done! + depends: + - s1 ``` ### 4. Execute the DAG @@ -247,42 +250,44 @@ dagu version ``` ## **Localized Documentation** -- [中文文档 (Chinese Documentation)](https://dagu.readthedocs.io/zh) -- [日本語ドキュメント (Japanese Documentation)](https://dagu.readthedocs.io/ja) + +- [中文文档 (Chinese Documentation)](https://dagu.readthedocs.io/zh) +- [日本語ドキュメント (Japanese Documentation)](https://dagu.readthedocs.io/ja) ## **Documentation** -- [Installation Instructions](https://dagu.readthedocs.io/en/latest/installation.html) -- ️[Quick Start Guide](https://dagu.readthedocs.io/en/latest/quickstart.html) -- [Command Line Interface](https://dagu.readthedocs.io/en/latest/cli.html) -- [Web User Interface](https://dagu.readthedocs.io/en/latest/web_interface.html) -- Writing DAG - - [Minimal DAG Definition](https://dagu.readthedocs.io/en/latest/yaml_format.html#minimal-dag-definition) - - [Running Arbitrary Code Snippets](https://dagu.readthedocs.io/en/latest/yaml_format.html#running-arbitrary-code-snippets) - - [Environment Variables](https://dagu.readthedocs.io/en/latest/yaml_format.html#defining-environment-variables) - - [Parameters](https://dagu.readthedocs.io/en/latest/yaml_format.html#defining-and-using-parameters) - - [Command Substitution](https://dagu.readthedocs.io/en/latest/yaml_format.html#using-command-substitution) - - [Conditional Logic](https://dagu.readthedocs.io/en/latest/yaml_format.html#adding-conditional-logic) - - [Environment Variables with Standard Output](https://dagu.readthedocs.io/en/latest/yaml_format.html#setting-environment-variables-with-standard-output) - - [Redirecting Stdout and Stderr](https://dagu.readthedocs.io/en/latest/yaml_format.html#redirecting-stdout-and-stderr) - - [Lifecycle Hooks](https://dagu.readthedocs.io/en/latest/yaml_format.html#adding-lifecycle-hooks) - - [Repeating Task](https://dagu.readthedocs.io/en/latest/yaml_format.html#repeating-a-task-at-regular-intervals) - - [Minimal DAG Definition](https://dagu.readthedocs.io/en/latest/yaml_format.html#minimal-dag-definition) - - [Running Sub-DAG](https://dagu.readthedocs.io/en/latest/yaml_format.html#running-sub-dag) - - [All Available Fields for a DAG](https://dagu.readthedocs.io/en/latest/yaml_format.html#all-available-fields-for-dags) - - [All Available Fields for a Step](https://dagu.readthedocs.io/en/latest/yaml_format.html#all-available-fields-for-steps) -- Example DAGs - - [Hello World](https://dagu.readthedocs.io/en/latest/examples.html#hello-world) - - [Conditional Steps](https://dagu.readthedocs.io/en/latest/examples.html#conditional-steps) - - [File Output](https://dagu.readthedocs.io/en/latest/examples.html#file-output) - - [Passing Output to Next Step](https://dagu.readthedocs.io/en/latest/examples.html#passing-output-to-next-step) - - [Running a Container Image](https://dagu.readthedocs.io/en/latest/examples.html#running-a-docker-container) - - [Making HTTP Requests](https://dagu.readthedocs.io/en/latest/examples.html#sending-http-requests) - - [JSON Processing](https://dagu.readthedocs.io/en/latest/examples.html#querying-json-data-with-jq) - - [Email](https://dagu.readthedocs.io/en/latest/examples.html#sending-email) -- [Configurations](https://dagu.readthedocs.io/en/latest/config.html) -- [Scheduler](https://dagu.readthedocs.io/en/latest/scheduler.html) -- [Docker Compose](https://dagu.readthedocs.io/en/latest/docker-compose.html) -- [REST API Documentation](https://app.swaggerhub.com/apis/YOHAMTA_1/dagu) + +- [Installation Instructions](https://dagu.readthedocs.io/en/latest/installation.html) +- ️[Quick Start Guide](https://dagu.readthedocs.io/en/latest/quickstart.html) +- [Command Line Interface](https://dagu.readthedocs.io/en/latest/cli.html) +- [Web User Interface](https://dagu.readthedocs.io/en/latest/web_interface.html) +- Writing DAG + - [Minimal DAG Definition](https://dagu.readthedocs.io/en/latest/yaml_format.html#minimal-dag-definition) + - [Running Arbitrary Code Snippets](https://dagu.readthedocs.io/en/latest/yaml_format.html#running-arbitrary-code-snippets) + - [Environment Variables](https://dagu.readthedocs.io/en/latest/yaml_format.html#defining-environment-variables) + - [Parameters](https://dagu.readthedocs.io/en/latest/yaml_format.html#defining-and-using-parameters) + - [Command Substitution](https://dagu.readthedocs.io/en/latest/yaml_format.html#using-command-substitution) + - [Conditional Logic](https://dagu.readthedocs.io/en/latest/yaml_format.html#adding-conditional-logic) + - [Environment Variables with Standard Output](https://dagu.readthedocs.io/en/latest/yaml_format.html#setting-environment-variables-with-standard-output) + - [Redirecting Stdout and Stderr](https://dagu.readthedocs.io/en/latest/yaml_format.html#redirecting-stdout-and-stderr) + - [Lifecycle Hooks](https://dagu.readthedocs.io/en/latest/yaml_format.html#adding-lifecycle-hooks) + - [Repeating Task](https://dagu.readthedocs.io/en/latest/yaml_format.html#repeating-a-task-at-regular-intervals) + - [Minimal DAG Definition](https://dagu.readthedocs.io/en/latest/yaml_format.html#minimal-dag-definition) + - [Running Sub-DAG](https://dagu.readthedocs.io/en/latest/yaml_format.html#running-sub-dag) + - [All Available Fields for a DAG](https://dagu.readthedocs.io/en/latest/yaml_format.html#all-available-fields-for-dags) + - [All Available Fields for a Step](https://dagu.readthedocs.io/en/latest/yaml_format.html#all-available-fields-for-steps) +- Example DAGs + - [Hello World](https://dagu.readthedocs.io/en/latest/examples.html#hello-world) + - [Conditional Steps](https://dagu.readthedocs.io/en/latest/examples.html#conditional-steps) + - [File Output](https://dagu.readthedocs.io/en/latest/examples.html#file-output) + - [Passing Output to Next Step](https://dagu.readthedocs.io/en/latest/examples.html#passing-output-to-next-step) + - [Running a Container Image](https://dagu.readthedocs.io/en/latest/examples.html#running-a-docker-container) + - [Making HTTP Requests](https://dagu.readthedocs.io/en/latest/examples.html#sending-http-requests) + - [JSON Processing](https://dagu.readthedocs.io/en/latest/examples.html#querying-json-data-with-jq) + - [Email](https://dagu.readthedocs.io/en/latest/examples.html#sending-email) +- [Configurations](https://dagu.readthedocs.io/en/latest/config.html) +- [Scheduler](https://dagu.readthedocs.io/en/latest/scheduler.html) +- [Docker Compose](https://dagu.readthedocs.io/en/latest/docker-compose.html) +- [REST API Documentation](https://app.swaggerhub.com/apis/YOHAMTA_1/dagu) ## **Running as a daemon** @@ -314,72 +319,72 @@ The YAML code below represents this DAG: ```yaml # Environment variables used throughout the pipeline env: - - DATA_DIR: /data - - SCRIPT_DIR: /scripts - - LOG_DIR: /log - # ... other variables can be added here + - DATA_DIR: /data + - SCRIPT_DIR: /scripts + - LOG_DIR: /log + # ... other variables can be added here # Handlers to manage errors and cleanup after execution handlerOn: - failure: - command: "echo error" - exit: - command: "echo clean up" + failure: + command: "echo error" + exit: + command: "echo clean up" # The schedule for the DAG execution in cron format # This schedule runs the DAG daily at 12:00 AM schedule: "0 0 * * *" steps: - # Step 1: Pull the latest data from a data source - - name: pull_data - command: "sh" - script: | - echo `date '+%Y-%m-%d'` - output: DATE - - # Step 2: Cleanse and prepare the data - - name: cleanse_data - command: echo cleansing ${DATA_DIR}/${DATE}.csv - depends: - - pull_data - - # Step 3: Transform the data - - name: transform_data - command: echo transforming ${DATA_DIR}/${DATE}_clean.csv - depends: - - cleanse_data - - # Parallel Step 1: Load the data into a database - - name: load_data - command: echo loading ${DATA_DIR}/${DATE}_transformed.csv - depends: - - transform_data - - # Parallel Step 2: Generate a statistical report - - name: generate_report - command: echo generating report ${DATA_DIR}/${DATE}_transformed.csv - depends: - - transform_data - - # Step 4: Run some analytics - - name: run_analytics - command: echo running analytics ${DATA_DIR}/${DATE}_transformed.csv - depends: - - load_data - - # Step 5: Send an email report - - name: send_report - command: echo sending email ${DATA_DIR}/${DATE}_analytics.csv - depends: - - run_analytics - - generate_report - - # Step 6: Cleanup temporary files - - name: cleanup - command: echo removing ${DATE}*.csv - depends: - - send_report + # Step 1: Pull the latest data from a data source + - name: pull_data + command: "sh" + script: | + echo `date '+%Y-%m-%d'` + output: DATE + + # Step 2: Cleanse and prepare the data + - name: cleanse_data + command: echo cleansing ${DATA_DIR}/${DATE}.csv + depends: + - pull_data + + # Step 3: Transform the data + - name: transform_data + command: echo transforming ${DATA_DIR}/${DATE}_clean.csv + depends: + - cleanse_data + + # Parallel Step 1: Load the data into a database + - name: load_data + command: echo loading ${DATA_DIR}/${DATE}_transformed.csv + depends: + - transform_data + + # Parallel Step 2: Generate a statistical report + - name: generate_report + command: echo generating report ${DATA_DIR}/${DATE}_transformed.csv + depends: + - transform_data + + # Step 4: Run some analytics + - name: run_analytics + command: echo running analytics ${DATA_DIR}/${DATE}_transformed.csv + depends: + - load_data + + # Step 5: Send an email report + - name: send_report + command: echo sending email ${DATA_DIR}/${DATE}_analytics.csv + depends: + - run_analytics + - generate_report + + # Step 6: Cleanup temporary files + - name: cleanup + command: echo removing ${DATE}*.csv + depends: + - send_report ``` ## **Motivation** @@ -396,7 +401,7 @@ There are many existing tools such as Airflow, but many of these require you to Dagu is a single command line tool that uses the local file system to store data, so no database management system or cloud service is required. DAGs are defined in a declarative YAML format, and existing programs can be used without modification. ----- +--- Feel free to contribute in any way you want! Share ideas, questions, submit issues, and create pull requests. Check out our [Contribution Guide](https://dagu.readthedocs.io/en/latest/contrib.html) for help getting started. diff --git a/docs/source/installation.rst b/docs/source/installation.rst index c62ed1823..44e6cbcda 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -11,13 +11,13 @@ Via Homebrew .. code-block:: bash - brew install yohamta/tap/dagu + brew install dagu-dev/brew/dagu Upgrade to the latest version: .. code-block:: bash - brew upgrade yohamta/tap/dagu + brew upgrade dagu-dev/brew/dagu Via Bash script ---------------