Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CI tests runtime #457

Merged
merged 7 commits into from
Aug 14, 2023
Merged

Improve CI tests runtime #457

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8be1022
Improve CI tests runtime
tatiana Aug 11, 2023
83d9e33
Move virtualenv test to expensive integration tests suite
tatiana Aug 11, 2023
7f1edb5
Fix issue with integration tests selector
tatiana Aug 11, 2023
8c4532e
Fix issue with expensive integration tests setup
tatiana Aug 14, 2023
c38b10c
Remove codecov change
tatiana Aug 14, 2023
58eea9f
Merge branch 'main' into improve-ci-test-time
tatiana Aug 14, 2023
3e58da4
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] Aug 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 36 additions & 2 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,13 @@ jobs:
with:
ref: ${{ github.event.pull_request.head.sha || github.ref }}

- uses: actions/cache@v3
with:
path: |
~/.cache/pip
.nox
key: unit-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('cosmos/__init__.py') }}

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
Expand Down Expand Up @@ -62,11 +69,9 @@ jobs:
airflow-version: ["2.3", "2.4", "2.5", "2.6"]
services:
postgres:
# Docker Hub image
image: postgres
env:
POSTGRES_PASSWORD: postgres
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
Expand All @@ -78,6 +83,12 @@ jobs:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha || github.ref }}
- uses: actions/cache@v3
with:
path: |
~/.cache/pip
.nox
key: integration-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('cosmos/__init__.py') }}

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
Expand Down Expand Up @@ -118,10 +129,33 @@ jobs:
matrix:
python-version: ["3.10"]
airflow-version: ["2.6"]

services:
postgres:
image: postgres
env:
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432

steps:
- uses: actions/checkout@v3
with:
ref: ${{ github.event.pull_request.head.sha || github.ref }}
- uses: actions/cache@v3
with:
path: |
~/.cache/pip
.nox
key: integration-expensive-${{ runner.os }}-${{ matrix.python-version }}-${{ matrix.airflow-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('cosmos/__init__.py') }}

- name: Checkout pull/${{ github.event.number }}
uses: actions/checkout@v3

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,15 +162,15 @@ test-integration = """pytest -vv \
--cov-report=xml \
--durations=0 \
-m integration \
-k 'not test_example_dag[example_cosmos_python_models]'
-k 'not (test_example_dag[example_cosmos_python_models] or test_example_dag[example_virtualenv])'
"""
test-integration-expensive = """pytest -vv \
--cov=cosmos \
--cov-report=term-missing \
--cov-report=xml \
--durations=0 \
-m integration \
-k 'test_example_dag[example_cosmos_python_models]'"""
-k 'test_example_dag[example_cosmos_python_models] or test_example_dag[example_virtualenv]'"""

[tool.pytest.ini_options]
filterwarnings = [
Expand Down