PYpelines

This is python-based fullstack CI/CD pipelines made for and by python community users.

Not groundbreaking, but very nice features:

full coverage of the best practices of not only our community, but of the world at large: linters that won't miss most bugs
presets for those who don't want to configure anything
flexible docker support: pipelines are compatible with any docker-based project
first class frontend support (for fullstack developers)
trivy security scanner
linters for docker files and for k8s manifests
badge with the number of lines of code
badge with the versions of your service on the preview and production environments: you will always see where the release is!

Contents:

Requirements
Stages
Project versioning and images tagging
Quickstart examples
Environment variables
- Skiping some parts of pipeline
Advanced
- Rules
- Frontend package managers
- Type coverage support
- Advanced stages
- static-analyze
- build
- test
- appsec
- scan
- publish
- image-release
- deploy
- gitlab-release
- e2e
Badges
FAQ

Requirements

Trunk-based development
Python-based project, frontend-based project (basically we contain frontend pipelines only for those who wish for fullstack development)
Semver versioning via git tags
Project structure:
- Folder chart in the root of your project, where helm chart is presented (it can be renamed with DEPLOY_HELM_CHART variable)
Some necessary environment variables

Stages

Any available preset contains a subset of all stages, available in the pipeline. More accurately, the following:

Stage	Description
`static-analyze`	Responsible for linting your code/Dockerfiles/Manifests
`build`	Used to build multiarch images via Dockerfile
`test`	Tests your builded image in `docker-compose` or right in cli
`appsec`	Application security scan
`scan`	Scan your image with sonar
`publish`	Publish project to Artifactory pypi
`image-release`	Release image with tag
`deploy`	Deploy to dev/prev/prod
`gitlab-release`	Create a gitlab release and set release-badge
`e2e`	Run e2e tests

Project versioning and images tagging

The tags automatically applied to images for their publication in Artifactory are determined through various stages and jobs.

Version from Git Tag: When a Git tagging event occurs, the corresponding version tag is used. This version tag (CI_COMMIT_TAG) is directly utilized for versioning the image, allowing a clear correlation of built images with code versions in the repository.

Short SHA of Commit: In addition to the version tag, an image can be tagged with the short hash of the commit (CI_COMMIT_SHORT_SHA), providing uniqueness and the ability to trace the image back to a specific change in the code.

Environment Tag: Depending on the environment to which deployment is being made (e.g., dev, test, preview, prod), an image can receive a corresponding environment tag, simplifying the identification of the image by purpose and used environment.

'Scanned' Tag: After successfully passing vulnerability scanning, an image is assigned the scanned tag, indicating that the image has been checked and deemed safe for use.

'Tested' Tag: Similarly, an image can receive a tested tag if it has successfully passed all tests. This is an additional confirmation of the image's quality and readiness for deployment.

All these tags are applied in accordance with the logic of the CI/CD pipeline tasks and are used for version management, tracking, and quality control of images that are then published in Artifactory. This approach ensures precise versioning, security, and traceability of images in the continuous integration and delivery process.

Quickstart examples

Recommended usage

The use of standardized pipelines is adapted for application through presets. It's recommended to use the whole preset. However, if for some reason you do not want to use the entire pipeline, you are free to assemble your own pipeline from individual primitives.

Usage of whole preset

You need to set all environment variables
Include preset inside .gitlab-ci.yml file.

# .gitlab-ci.yml
variables:
  ENABLE_ISOLATION: "true"

include:
  - project: "python-community/pypelines"
    file: "presets/backend--v1.yml"

By default, we don't run tests in branches because it motivates us to use the practice of continuous integration (we integrate our code into the main branch many times a day) rather than continuous isolation (in this practice you live very long in branches). However, most developers are used to seeing tests in their branches, so setting ENABLE_ISOLATION to true gives this result.

Usage some parts of preset

Use parts of preset

include:
  - project: "python-community/pypelines"
  - file: "templates/v1/deploy.yml"

deploy-prod-only:
  extends: .deploy-prod

gitlab-release-prod-only:
  extends: .gitlab-release
  variables:
    RELEASE_SERVER: production

Use rules and scripts by reference:

include:
  - project: "python-community/pypelines"
    file: "_base/rules.yml "
  - project: "python-community/pypelines"
    file: "_base/jobs.yml "

notify-team-on-new-tag:
  before_script:
    - !reference [.default-docker-cli-job, before_script]
  script:
    - python -m <some notification app run>
  rules:
    - !reference [.default-rules, tag-rule]

Creating complex tests

We assume to run simple tests in our presets. In case you need something more complex you can use a compose file. This file meant equally for local development and also it will be used in pipelines. To do so you need create file like this:

#/docker-compose.yml
services:
  application: # this is standard name, you MUST define it like this
    build: # this section will be automatically removed by pypelines and replaced with image section
      context: .
    depends_on:
      - postgres
      - redis
      - rabbitmq
    restart: always
    volumes:
      - .:/srv/www/
    ports:
      - "8000:8000"

  # Remove unnecessary blocks or add images of systems used in your project
  postgres:
    image: $ARTIFACTORY_HOST/python-community-docker/techimage/postgres/postgres:15.2-alpine3.17
    environment:
      POSTGRES_DB: SOME_DB_NAME
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    ports:
      - "5432:5432"

  redis:
    image: $ARTIFACTORY_HOST/python-community-docker/techimage/redis/redis:6.2.11-alpine3.17
    ports:
      - "6379:6379"
    environment:
      - ALLOW_EMPTY_PASSWORD=yes

  rabbitmq:
    image: $ARTIFACTORY_HOST/python-community-docker/techimage/rabbitmq/rabbitmq:3.11.13-management
    hostname: rabbitmq
    restart: always
    environment:
      - RABBITMQ_DEFAULT_USER=rmuser
      - RABBITMQ_DEFAULT_PASS=rmpassword
    ports:
      - 15672:15672

# .gitlab-ci.yml
include:
  - project: "python-community/pypelines"
    file: "templates/v1/build.yml"

  - project: "python-community/pypelines"
    file: "_base/jobs.yml"

deploy-dev:
  extends: .deploy-dev

deploy-docker-compose:
  before_script:
    - !reference [.default-docker-cli-job, before_script]
  script:
    - docker compose run postgres
  needs:
    - deploy-dev

Gitlab Integrations

Also this package gives integration with gitlab codequality(https://docs.gitlab.com/ee/ci/testing/code_quality.html) and container scanning(https://docs.gitlab.com/ee/user/application_security/container_scanning/).

Jobs with codequality reports support:

ruff
mypy
hadolint

Jobs with container scanning report support:

trivy

If you want to add any utility for static analyze or container scanning - check if it supports gitlab integration and may be integrated. To enable integration with gitlab you need to store report and pass it via artifacts:

# for codequality(take in mind that report must be stored in this path)
artifacts:
  reports:
    codequality: $CODEQUALITY_REPORT

# for container scanning(take in mind that report must be stored in this path)
artifacts:
  reports:
    container_scanning: reports/$CONTAINER_SCANNING_REPORT

Used utilities

Under the hood this pipeline uses those utilities.

Utility	Description
`Node`	Frontend purposes
`Docker`	Run different images on the spot.
`Ruff`	Lint and validate formatting of python code.
`Mypy`	Static analyzer for python code.
`Trivy`	Vulnerability scanner for python code.
`Hadolint`	Lint Dockerfiles for best practices.
`Kubescore`	Lint kube manifests for best practices.
`Pygount`	Count lines of code.
`Anybadge`	Adding lines of code badge.
`Kaniko`	Build Docker images using Dockerfile.
`yq`	Substitute to `docker-compose` files during test stage.
`Appscreener`	Appsec scan image.
`Sonar`	Scan image.
`Jfrog-cli`	Set and check properties on images in the artifactory.
`Skopeo`	Copy images during release stage.
`Tiny Releaser`	Making releases in gitlab CI.

Environment variables

This pipeline has the following environment variables, which should be configured before first run:

Note: If the variable isn't set, corresponding stage will be skipped during pipeline execution. If required variable that used in multiple stages isn't set, whole pipeline may fail.

Variable	Required	Stage	Description
`ARTIFACTORY_USER`	yes	multiple	Login for the service user, which will be used to push/pull artifacts to/from the Artifactory
`ARTIFACTORY_PASSWORD`	yes	multiple	Password for the service user, which will be used to push/pull artifacts to/from the Artifactory
`DOCKER_TEAM_REGISTRY_PREFIX`	yes	multiple	Docker team registry in the Artifactory for built Docker images
`DEPLOY_HELM_CHART`	-	multiple	Directory for helm charts
`PROJECT_SLUG`	-	multiple	Must be set if you need to specify subdir of your project. Your project path: `root-group/child-group/grandchild-group/project-name` Without this var you will get path of artifactory REGISTRY as `$ARTIFACOTRY_HOST/$DOCKER_TEAM_REGISTRY_PREFIX/project-name/` But with `PROJECT_SLUG="/child-group/grandchild-group"` It would be: `$ARTIFACTORY_HOST/$DOCKER_TEAM_REGISTRY_PREFIX/child-group/grandchild-group/project-name/` Note: Don't forget "/" at the start of the var!
`ENABLE_ISOLATION`	-	multiple	if set to "true", then build, test and scan jobs will be run on merge request event
`NPMRC_ADDITIONAL_SCOPE`	-	multiple	With this variable you can add additional sections for the .npmrc file (dynamically created in CI/CD)
`COREPACK_NPM_REGISTRY`	-	multiple	With this variable you can modify where corepack will search for package managers
`NPM_INSTALL_EXTRA_ARGS`	-	multiple	Arguments for installing packages via npm
`PNPM_INSTALL_EXTRA_ARGS`	-	multiple	Arguments for installing packages via pnpm
`YARN_INSTALL_EXTRA_ARGS`	-	multiple	Arguments for installing packages via yarn
`SEVERITY_COMMAND`	-	static-analyze	Filter by severity for trivy (https://aquasecurity.github.io/trivy/v0.48/docs/configuration/filtering/#by-severity)
`CODEQUALITY_REPORT`	-	static-analyze	File path for storing gitlab codequality report(about)
`CONTAINER_SCANNING_REPORT`	-	static-analyze	File path for storing gitlab container scanning report(about)
`RUFF_VERSION`	-	static-analyze	Specify ruff version to use in pipeline
`MYPY_VERSION`	-	static-analyze	Specify mypy version to use in pipeline
`SECURITY_TEAM_ID`	-	appsec	If you want automatic AppSec in your pipeline, you must fill this with you team id.
`SONAR_TOKEN`	-	scan	Token for logging in Sonar (need to took from here: https://some.sonar.ru/)
`SONAR_TEAM`	-	scan	Sonar project for the team (should be presented here: https://some.sonar.ru/)
`TEAM_PYPI_REPO`	-	publish	Pypi team name for package publishing. Required only in `presets/python-package--v1.yml` pipeline
`DEPLOY_NAMESPACE` or `DEPLOY_NAMESPACE_<ENV>`	-	deploy	Namespace, where deploy should be performed. It is possible to set one general variable for all available environments, or multiple different variables for specific environments. ENV should be one of the following: dev, prev, prod.
`KUBE_CONTEXT` or `KUBE_CONTEXT_<ENV>`	-	deploy	Gitlab agent for the namespace, where deploy should be performed. It is possible to set one general variable for all available environments, or multiple different variables for specific environments. ENV should be one of the following: dev, prev, prod.
`ASSET_URL`	-	deploy	Link to artifactory asset for tiny-releaser (https://gitlab.ru/python-community/pypi/tiny-releaser)
`GENERIC_GITLAB_TOKEN`	-	gitlab-release	Group or Project token, Maintainer role is required to create badges
`E2E_PROJECT`	-	e2e	Path to the repository, where E2E tests are presented

Skiping some parts of pipeline

If you need to skip some jobs or stages you can use skipping variables. The following variables must be set to True to skip some stages of this pipeline:

| Variable | Description | | -------------------------- | ---------------------------------- | ------ | | SKIP_TRIVY | Skip trivy static analysis | | SKIP_HADOLINT | Skip hadolint static analysis | | SKIP_KUBESCORE | Skip kube score static analysis | | SKIP_BUILD_ARM | Skip kaniko-build-arm stage | | SKIP_TESTS | Skip tests execution | | APPSEC_SKIP | Skip appsec stage | | SKIP_AUTO_SEMVER | Skip auto semver version bumping | | SKIP_GITLAB_RELEASE | Skip of creating gitlab release | | SKIP_RELEASE_PAGE_UPDATE | Skip of updating releases infopage | !TO DO |

Advanced

If you want to understand, what's going on under the hood - this section is right for you. Here we will analyze how the backend preset works on the inside.

Rules

In this pipeline we have a lot of different rules, because we wanted to be blazingly fast during our development process. You can see the full list of rules here. In this advanced section of the pipeline documentation, it will be marked when jobs from different stages are to be run.

The rules are exactly like this because motivation of this pipeline is to provide continuous integration and not continuous isolation or ci theatre

If you are a continuous isolation enjoyer, then this pipeline has something special just for ya. Behold! Continuous isolation mode, that can be easily enabled by setting ENABLE_ISOLATION variable to "true". This mode turned off by default.

Frontend package managers

This pipeline supports usage of this package managers:

This functionality is provided by corepack(https://nodejs.org/api/corepack.html). To use yarn or pnpm in pipelines(for local develop too) you need to specify packageManager in package.json file. How to do this you can read here. If no package manager specified - npm will be used by default.

Note: If you want to use package manager in your jobs to install some packages or lint or smth else - you should extend .configure-package-manager job. Also prefered way to use $PACKAGE_MANAGER variable and not direct package manager name. So linting with this variable will look like this:

$PACKAGE_MANAGER run lint

For each package manager we have separate variables for installing packages. You can also modify them for your needs.

Type coverage support

For frontend packages we have support of type coverage badges.

To use this feature you should modify lint-frontend job to generate svg badge file. By default this file should have name , but you can change this by modifying env TYPE_COVERAGE_BADGE. You can use example below for using this feature for your project:

Example:

At first you need to configure package.json file to generate type coverage report. We recommend you to use type-coverage package for this purpose: https://github.com/plantain-00/type-coverage

Example uses pnpm package manager but you can use your own.

"scripts": {
  "lint": "... && pnpm run lint:typecheck",
  "lint:typecheck": "pnpm dlx type-coverage | tee type-coverage.json",
},

Then you need to configure parameters for type-coverage. You can view all possible params here https://github.com/plantain-00/type-coverage?tab=readme-ov-file#arguments. You can do this in package.json or specify directly in lint:typecheck.

"typeCoverage": {
  "project": "tsconfig.json",
  "is": 100,
  "detail": true,
  "strict": true,
  "reportSemanticError": true,
  "jsonOutput": true
},

Then you need generate badge from type-coverage.json report file in lint-frontend job. You can do this by modifying .gitlab-ci.yml:

lint-frontend:
  extends: .lint-frontend
  script:
    - !reference [.lint-frontend, script]
    - ./type-coverage-badge.sh

You can use this script to generate badge or write your own:

if [ $# -eq 0 ]; then type_coverage_report='type-coverage.json'
else type_coverage_report=$1; fi

if ! [ -f $type_coverage_report ]; then
  echo "Can't build badge, because $type_coverage_report does not exist."
  exit 0
fi

type_coverage_percent=$(awk '/"percent":/ { print substr($2, 1, length($2)-1)}' $type_coverage_report)
int_type_coverage_percent=${type_coverage_percent%.*}

if [ $int_type_coverage_percent -eq 100 ]; then badge_color="green"
elif [ $int_type_coverage_percent -ge 90 ]; then badge_color="yellowgreen"
elif [ $int_type_coverage_percent -ge 70 ]; then badge_color="yellow"
elif [ $int_type_coverage_percent -ge 50 ]; then badge_color="orange"
else badge_color="red"; fi

npx badge-maker --label "type coverage" --message "$type_coverage_percent%" --color $badge_color > type-coverage-badge.svg

And then you need to add badge support for your repository(badges)

Advanced stages

static-analyze

Here we run the following code analysis tools. Most of this tools are required, but some can be skipped with the flag. For the required tools you will see "-" as the skip flag name. In this table you can see:

name of the tool
name of the flag, that is responsible for this tool to be run or not
presets, that using this tool

Jobs from this stage will be run only on merge requests.

Tool	Flag	Presets
`ruff`	-	Backend, Package
`mypy`	-	Backend, Package
`trivy`	SKIP_TRIVY	Backend, Package
`hadolint`	SKIP_HADOLINT	Backend
`kube-score`	SKIP_KUBESCORE	Backend
`lines-of-code-badge`	-	Backend, Frontend, Package
`lint-frontend`	-	Frontend

Template

build

At this stage image will be multiarch built and published to the artifactory, you have provided. All images and manifests will be stored in ci project folder. Unfortunately, none can be skipped out of the box at this stage. In this pipeline we are using kaniko because of it performance.

This is your artifactory project tree, after build stage has passed. Here DOCKER_TEAM_REGISTRY_PREFIX, PROJECT_SLUG, CI_PROJECT_NAME - names of variables, you have to set in advance.

DOCKER_TEAM_REGISTRY_PREFIX
└── PROJECT_SLUG
    └── CI_PROJECT_NAME
        └── ci
            ├── 027c3e98
            ├── 027c3e98-linux-amd64
            └── 027c3e98-linux-arm64

Template

test

This stage provides only one job test-compose, that can be skipped with the flag SKIP_TESTS. Job will launch a docker compose, that you have to provide. This job creates an artifact, that can be used to make a coverage badge.

After successful testing if commit is on master/main/hotfix branches, then "tested" property will be added to image.

Template

appsec

This stage provide application security scan via Appsec AppScreener. Job need some environment variables to be set: APPSCREENER_TOKEN and can be also skipped via APPSEC_SKIP.

Template

scan

Scanning is happening here, currently, there is only one tool: SonarQube. It can be skipped with the flag SKIP_SONAR.

After successful testing if commit is on master/main/hotfix branches, "scanned" property will be added to image.

Template

publish

At this stage, job will publish project to Artifactory PyPi. This job will build project via poetry build for picked commit and then publish to team PyPi directory in Artifactory. You also need to specify team name TEAM_PYPI_REPO for publishing.

Template

image-release

This stage may consist of two jobs(verify-artifactory-properties and image-release).

If pipeline is runned on master/main/hotfix branches or if CI_COMMIT_TAG is specified for commit and is semver-like, we will have two jobs in stage. Job verify-artifactory-properties will will search image with tested and scanned properties. If no such image found - pipeline will fail. After this job image-release will try to copy image to the release project folder.
If pipeline is runned on merge requests with enabled isolation(ENABLE_ISOLATION), then stage will consist of only one manual job: image-release.
In other cases image-release stage will be empty.

Some words about work of image-release job:

Imagine, you create a new tag 1.0.0 with commit hash 027c3e98.

You will have the following tree, after this stage pass.

DOCKER_TEAM_REGISTRY_PREFIX
└── PROJECT_SLUG
    └── CI_PROJECT_NAME
        ├── release
        │   └── 1.0.0
        └── ci
            ├── 027c3e98
            ├── 027c3e98-linux-amd64
            └── 027c3e98-linux-arm64

This is not the only use-case. Images will be released even without tags then name of the release will be CI_COMMIT_SHORT_SHA.

DOCKER_TEAM_REGISTRY_PREFIX
└── PROJECT_SLUG
    └── CI_PROJECT_NAME
        ├── release
        │   └── 027c3e98
        └── ci
            ├── 027c3e98
            ├── 027c3e98-linux-amd64
            └── 027c3e98-linux-arm64

Template

deploy

This stage contains different jobs, that uses different gitlab environments. For example, to deploy to prod you should define a KUBE_CONTEXT and DEPLOY_NAMESPACE in your environment, or you can set KUBE_CONTEXT_PROD and DEPLOY_NAMESPACE_PROD globally.

Here we also use artifactory properties, to ignore certain artifacts during cleanup process. After deploy on prod was successful, release-prod property will be unset from all images inside release folder and than set to the image that was just released.

Rules for this stage are pretty complex, so please, see it yourself here.

In isolation mode, auto semver job will be triggered after merging to default branch, which bumps version and creates a new tag

After deploy the following environment variables are passed to containers:

CI_COMMIT_TAG - commit tag if exists
CI_COMMIT_SHA - commit hash
CI_COMMIT_REF_NAME - branch name
CI_PROJECT_NAME - gitlab repo name

Template

gitlab-release

This stage will automatically create a new gitlab release and put a link to the released artifact. Is also creates a release badge, that clearly indicates release version. Jobs require GENERIC_GITLAB_TOKEN with write rules for badge and release creation. There are two jobs here, you can skip all of them. Rules are not so obvious and requires some attention.

Job	Flag
`gitlab-release`	SKIP_GITLAB_RELEASE
`gitlab-release-update-prod`	SKIP_GITLAB_RELEASE or SKIP_RELEASE_PAGE_UPDATE

Template

e2e

Here downstream pipeline technique is used. You have to provide E2E_PROJECT to run the pipeline there. Jobs will be run if E2E_PROJECT is provided and CI_COMMIT_TAG is semver-like.

Template

P.S.
If you feel like you misunderstood something, feel free to read the source code. There's a small description nearby every job, that will help you to understand the behavior and requirements.

Badges

If you are using one of presets, or inherited from some jobs, you can add badges to your project. Here job name - is the one, that provides some artifact after its execution, that can be used in badges. If there is no job name, then you can add badge without using any job.

Job name	Badge
`.lint-frontend`
`.lines-of-code-badge`
`.test-python-ci`/`.test-compose-ci`
`.gitlab-release`
if you use static analyze jobs

FAQ

Why do we use docker compose in tests? Docker compose is very flexible, so you can decide which services do you need in your particular case. Also, we cant hard-code services for any imaginable dependency in the world.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

PYpelines

Requirements

Stages

Project versioning and images tagging

Quickstart examples

Recommended usage

Usage of whole preset

Usage some parts of preset

Creating complex tests

Gitlab Integrations

Used utilities

Environment variables

Skiping some parts of pipeline

Advanced

Rules

Frontend package managers

Type coverage support

Advanced stages

static-analyze

build

test

appsec

scan

publish

image-release

deploy

gitlab-release

e2e

Badges

FAQ

About

Releases

Packages

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
_base		_base
presets		presets
scripts		scripts
templates/v1		templates/v1
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
README.md		README.md
logo.png		logo.png
pyproject.toml		pyproject.toml

insani7y/moscow-python-conf-2024

Folders and files

Latest commit

History

Repository files navigation

PYpelines

Requirements

Stages

Project versioning and images tagging

Quickstart examples

Recommended usage

Usage of whole preset

Usage some parts of preset

Creating complex tests

Gitlab Integrations

Used utilities

Environment variables

Skiping some parts of pipeline

Advanced

Rules

Frontend package managers

Type coverage support

Advanced stages

static-analyze

build

test

appsec

scan

publish

image-release

deploy

gitlab-release

e2e

Badges

FAQ

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages