Built Objects from Docker Build Don't Carry Over

[aidanngill]

My aim is to use pytest-celery for my tests for a Django project. The issue that I am facing is that a folder I create in my worker's Dockerfile is no longer present when the worker runs.

FROM python:3.12.4-bullseye AS builder

RUN pip install poetry==1.8.3

ENV POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    POETRY_CACHE_DIR=/tmp/poetry_cache

WORKDIR /build

COPY pyproject.toml poetry.lock ./

RUN poetry install --without dev --no-root && rm -rf $POETRY_CACHE_DIR

# ---------------------------------------------------------------------------- #

FROM python:3.12.4-slim-bullseye AS runtime

# Create a user to run the worker
RUN adduser --disabled-password --gecos "" test_user

# Set arguments
ARG CELERY_LOG_LEVEL=INFO
ARG CELERY_WORKER_NAME=celery_dev_worker
ARG CELERY_WORKER_QUEUE=celery
ENV LOG_LEVEL=$CELERY_LOG_LEVEL
ENV WORKER_NAME=$CELERY_WORKER_NAME
ENV WORKER_QUEUE=$CELERY_WORKER_QUEUE
ENV DJANGO_SETTINGS_MODULE=pterodactyl_billing.settings.testing

EXPOSE 5678

# Install packages
RUN mkdir -p /build/.venv

ENV VIRTUAL_ENV=/build/.venv \
    PATH="/build/.venv/bin:$PATH"

COPY --from=builder ${VIRTUAL_ENV} ${VIRTUAL_ENV}

# Switch to the test_user
USER test_user

WORKDIR /app

# Start the celery worker
CMD celery -A pterodactyl_billing worker --loglevel=$LOG_LEVEL -n $WORKER_NAME@%h -Q $WORKER_QUEUE

In my Dockerfile I use Poetry to install my dependencies to /build, away from /app and /src as those are overwritten later when the container is created.

My goal is to then use this /build folder as a virtual environment for Celery to use. However, when I run pytest it crashes, stating that Django is not installed. To try and solve this I explicitly stated that the /build/.venv/bin/celery binary should be used, but this reveals that this path is completely absent from the image.

Is there something I'm missing about the way pytest-celery creates images? If I build the worker image myself and run bash on it the directory exists there.

My worker creation script is also shown below.

from typing import Any
import celery
import os
from typing import Any
import celery
from pytest_docker_tools import build
from pytest_docker_tools import container
from pytest_docker_tools import fxtr
from pytest_celery import CeleryWorkerContainer
from pytest_celery import defaults


class DjangoWorkerContainer(CeleryWorkerContainer):
    @property
    def client(self) -> Any:
        return self

    @classmethod
    def version(cls) -> str:
        return celery.__version__

    @classmethod
    def log_level(cls) -> str:
        return "INFO"

    @classmethod
    def worker_name(cls) -> str:
        return "django_tests_worker"

    @classmethod
    def worker_queue(cls) -> str:
        return "celery"


worker_image = build(
    path=".",
    tag="aidanngill/pbp-worker:latest",
    dockerfile="billing/tests/worker/Dockerfile",
    buildargs=DjangoWorkerContainer.buildargs(),
)


default_worker_container = container(
    image="{worker_image.id}",
    ports=fxtr("default_worker_ports"),
    environment=fxtr("default_worker_env"),
    network="{default_pytest_celery_network.name}",
    volumes={
        # Volume: Worker /app
        "{default_worker_volume.name}": defaults.DEFAULT_WORKER_VOLUME,
        # Mount: source
        os.path.abspath(os.getcwd()): {
            "bind": "/src",
            "mode": "rw",
        },
    },
    wrapper_class=DjangoWorkerContainer,
    timeout=15,
)

...and the error I'm facing when running pytest.

--------------------------------------------------------------------------------------------- Captured stdout setup ---------------------------------------------------------------------------------------------
Creating network pytest-782d00c2-2621-4f70-aa0c-3e774ccd5755
Waiting for container to be ready..........RabbitMQContainer::jolly_brahmagupta is ready.

Waiting for container to be ready.RedisContainer::silly_shaw is ready.

Creating volume pytest-d887b353-3ff8-498f-ac4f-6ed926a0f984
Waiting for container to be ready.Waiting for CeleryWorkerContainer::festive_leavitt to get ready..............................
Waiting for CeleryWorkerContainer::festive_leavitt to get ready..............................
Waiting for CeleryWorkerContainer::festive_leavitt to get ready..............................

------------------------------------------------------------------------------------------------ festive_leavitt ------------------------------------------------------------------------------------------------
Usage: celery [OPTIONS] COMMAND [ARGS]...
Try 'celery --help' for help.

Error: Invalid value for '-A' / '--app':
Unable to load celery application.
While trying to load the module app the following error occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/bin/celery.py", line 58, in convert
    return find_app(value)
  File "/usr/local/lib/python3.10/site-packages/celery/app/utils.py", line 383, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/usr/local/lib/python3.10/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
    return imp(module, package=package)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/app.py", line 13, in <module>
    from billing.tasks import *
  File "/app/billing/tasks.py", line 4, in <module>
    from django.conf import settings
ModuleNotFoundError: No module named 'django'

I also see here that it's using python3.10 instead of python3.12 which is built into the Dockerimage?

Any help is appreciated, thanks.

[suspiciousRaccoon]

hi! /ᐠ. ̫ .ᐟ\ฅ

I've found similar problems when trying to integrate pytest-celery with uv. After too much time trying random stuff I've come to the conclusion that pytest-celery doesn't like virtual enviroments. Even if the image spins containers correctly in the cli, when trying with pytest it isn't able to find the venv. Maybe because the docker-py library uses the old legacy builder, I found problems with Buildkit when I tried using the docker files in this repo, but I don't know if that's the culprit.

I tried making the django example work with venvs by simply creating one and installing everything there but it didn't work.
The default_worker_env also contains a PYTHONPATH variable that points to /app and there's also the "bind": "/src" thing, which might be part of the problem, but after playing a bit with the config I've given up on making venvs work.

My solution boiled down to exporting all the dependencies into a requirements.txt file and then installing that globally with pip.

Here is the docker image I'm using for reference:

FROM python:3.12-bookworm

# Set up uv
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

# Create a user to run the worker
RUN adduser --disabled-password --gecos "" test_user

# Install system dependencies
RUN apt-get update && apt-get install -y build-essential git

# Set arguments
ARG CELERY_LOG_LEVEL=INFO
ARG CELERY_WORKER_NAME=celery_dev_worker
ARG CELERY_WORKER_QUEUE=celery
ENV LOG_LEVEL=$CELERY_LOG_LEVEL
ENV WORKER_NAME=$CELERY_WORKER_NAME
ENV WORKER_QUEUE=$CELERY_WORKER_QUEUE

EXPOSE 5678

# Install packages
WORKDIR /src

# Copy dependency files before installing dependencies
COPY uv.lock pyproject.toml /src/

# Install dependencies
RUN uv export --no-hashes --output-file requirements.txt
RUN pip install --no-cache-dir --upgrade pip
RUN pip install -r ./requirements.txt

# Switch to the test_user
USER test_user

# Start the celery worker
CMD celery -A proj worker --loglevel=$LOG_LEVEL -n $WORKER_NAME@%h -Q $WORKER_QUEUE

I used this with the django example and all tests passed. I only added the pyproject.toml with uv and changed the path in the build fixture to the new dockerfile.

I don't know where the problem is, but this works ¯_(ツ)_/¯

It'd be great if an example with venvs was added to the project.

For reference this is the dockerfile that I tried to use for venvs with the django example:

FROM python:3.11-bookworm

# Create a user to run the worker
RUN adduser --disabled-password --gecos "" test_user

# Install system dependencies
RUN apt-get update && apt-get install -y build-essential git

# Set arguments
ARG CELERY_LOG_LEVEL=INFO
ARG CELERY_WORKER_NAME=celery_dev_worker
ARG CELERY_WORKER_QUEUE=celery
ENV LOG_LEVEL=$CELERY_LOG_LEVEL
ENV WORKER_NAME=$CELERY_WORKER_NAME
ENV WORKER_QUEUE=$CELERY_WORKER_QUEUE

EXPOSE 5678

# Install packages
WORKDIR /src

RUN python -m venv venv

ENV PATH="/src/venv/bin:$PATH"

COPY --chown=test_user:test_user requirements.txt .
RUN pip install --no-cache-dir --upgrade pip
RUN pip install -r ./requirements.txt

# Switch to the test_user
USER test_user

# Start the celery worker
CMD celery -A proj worker --loglevel=$LOG_LEVEL -n $WORKER_NAME@%h -Q $WORKER_QUEUE

Which raises:
pytest_docker_tools.exceptions.ContainerNotReady: Timeout while waiting for container to be ready
And has this in the output:
/bin/sh: 1: celery: not found

[suspiciousRaccoon]

I'm pinging @Nusnus just in case, maybe this should be in the documentation?