Modular Pypeline Library

What is MPyL?

MPyL stands for Modular Pipeline Library (in Python).

This library is loosely based on the principles described in https://www.jenkins.io/blog/2019/01/08/mpl-modular-pipeline-library/ but completely independent of Jenkins or any other CI/CD platform.

🔖 Documentation

Detailed, complete, searchable documentation can be found at https://vandebron.github.io/mpyl

📚 Principles

MPyL is built with the following principles in mind.

• Not a platform It is not tied to any CI/CD platform. MPyL is a library, not a framework. It is not a runner, not a scheduler, and it doesn't have a GUI. It can be plugged into any CI/CD platform. Execution flows for Jenkins or Dagster are included as an example.
• Minimal footprint: It is self-contained and has very few external dependencies (e.g. Git, Docker). It can be run anywhere, also on your local machine.
• Accessible and maintainable
  • Written in Python, the most widely adopted scripting language with an extensive amount of client libraries relevant to CI/CD.
  • Strongly typed: MyPy type hints and schemas for all YAML files. Clearly defined interfaces for inputs and outputs of steps.
  • Focus on short feedback loop. Unit testable and everything can be run locally.
• Self documented by docstrings in the code, descriptions in the schemas and explanations in the CLI.
• Python as YAML templating engine Attempts to mix markup and logic tend to have a worst of both worlds outcome. MPyL uses YAML strictly for configuration. Where YAML needs to produced dynamically, like for CRDs via HELM charts, it is generated by Python code. The produced YAML is validated against the corresponding schemas.
• Extensible Support for the most common use cases (e.g. Docker, Helm, Kubernetes, AWS, etc.) is built-in. But it is easy to extend with your own mpyl.steps.

💻 Technologies

Requirements

The following technologies are expected to be present on the local OS:

• Python >= 3.9
• Pip >= 23.0.1
• Docker > 20
• Docker compose installed as plugin (docker compose version) >= v2.2.3
• Git SCM

Bundled

MPyL is extensible and has a minimal footprint. Having said that, batteries for the following technologies are included.

Deployment

• K8S mpyl.steps.deploy.kubernetes
• Helm mpyl.steps.deploy.k8s.helm