Specifying automated containers for airflow helm

[austinbv]

Describe the bug

We are running into a weird issue with the HelmOperator - We are trying to configure the automated deployments of custom images in the Airflow Helm Chart in the gitsync containers and initContainers. We have added annotations to our helm release and it looks like the below. We are not seeing any logs about the initContainer or gitSync and new images are not detected.

kind: HelmRelease
metadata:
  name: airflow
  namespace: airflow
  annotations:
    fluxcd.io/automated: true

    filter.fluxcd.io/airflow: glob:airflow-dev-*

    filter.fluxcd.io/initContainer: glob:git-*
    repository.fluxcd.io/initContainer: dags.initContainer.image.repository
    tag.fluxcd.io/initContainer: dags.initContainer.image.tag

    filter.fluxcd.io/gitSync: glob:git-*
    repository.fluxcd.io/gitSync: dags.git.gitSync.image.repository
    tag.fluxcd.io/gitSync: dags.git.gitSync.image.tag
spec:
  releaseName: airflow
  chart:
    repository: https://kubernetes-charts.storage.googleapis.com
    version: 7.1.x
    name: airflow
  values:
    airflow:
      image:
        repository: ****.dkr.ecr.****.amazonaws.com/****
        tag: airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604
    dags:
      initContainer:
        enabled: true
        image:
          repository: ****.dkr.ecr.****.amazonaws.com/****
          tag: git-41d01deba5c2a08edd5e006fddd8fe8497539a73
      git:
        gitSync:
          enabled: true
          image:
            repository: ****.dkr.ecr.****.amazonaws.com/****
            tag: git-41d01deba5c2a08edd5e006fddd8fe8497539a73

Additional context

• Helm Operator version: 1.1.0
• Kubernetes version: v1.16.8-eks-e16311
• Git provider: github
• Helm repository provider: stable/airflow

[austinbv]

I have added an issue (fluxcd/flux#3143) to the main flux repo. It seems to have the code but I know this is a feature with the CRD so I wasn't sure where's correct to file the issue.

[austinbv]

Running fluxctl list-images shows clear output that flux is aware the image has been deployed

airflow:statefulset/airflow-worker        git-sync           *****.dkr.ecr.****.amazonaws.com/*****
                                                             |   git-1cd41db74183568c674ea1b84a2db2d43043b27e                   17 Jun 20 17:51 UTC
                                                             '-> git-41d01deba5c2a08edd5e006fddd8fe8497539a73                   17 Jun 20 16:51 UTC
                                                                 airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604           17 Jun 20 16:32 UTC
                                                                 git-63b3d5ed649cb3e41611c8fc5eb8dfc583433604                   17 Jun 20 16:30 UTC
                                                                 dev-4a48bcbdf3687f365f2fb0dc5433f38e203021b3                   17 Jun 20 15:03 UTC
                                                                 dev-3e7414cf7bc5080546e8883826d4c9a38d890ce5                   16 Jun 20 16:50 UTC
                                                                 dev-08fa3d8b68641f941fc7925d3c194e047c8c1bb9                   16 Jun 20 16:36 UTC
                                                                 dev-9a5a828665509d0b14e39629a8ff173b1bdf7470                   02 Jun 20 17:55 UTC
                                                                 prod-efb1dc35d18131d35415d4d884fed7da18d9f569                  02 Jun 20 17:55 UTC
                                          airflow-worker     *****.dkr.ecr.****.amazonaws.com/*****
                                                             |   git-1cd41db74183568c674ea1b84a2db2d43043b27e                   17 Jun 20 17:51 UTC
                                                             |   git-41d01deba5c2a08edd5e006fddd8fe8497539a73                   17 Jun 20 16:51 UTC
                                                             '-> airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604           17 Jun 20 16:32 UTC
                                                                 git-63b3d5ed649cb3e41611c8fc5eb8dfc583433604                   17 Jun 20 16:30 UTC
                                                                 dev-4a48bcbdf3687f365f2fb0dc5433f38e203021b3                   17 Jun 20 15:03 UTC
                                                                 dev-3e7414cf7bc5080546e8883826d4c9a38d890ce5                   16 Jun 20 16:50 UTC
                                                                 dev-08fa3d8b68641f941fc7925d3c194e047c8c1bb9                   16 Jun 20 16:36 UTC
                                                                 dev-9a5a828665509d0b14e39629a8ff173b1bdf7470                   02 Jun 20 17:55 UTC
                                                                 prod-efb1dc35d18131d35415d4d884fed7da18d9f569                  02 Jun 20 17:55 UTC
                                          git-clone          *****.dkr.ecr.****.amazonaws.com/*****
                                                             |   git-1cd41db74183568c674ea1b84a2db2d43043b27e                   17 Jun 20 17:51 UTC
                                                             '-> git-41d01deba5c2a08edd5e006fddd8fe8497539a73                   17 Jun 20 16:51 UTC
                                                                 airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604           17 Jun 20 16:32 UTC
                                                                 git-63b3d5ed649cb3e41611c8fc5eb8dfc583433604                   17 Jun 20 16:30 UTC
                                                                 dev-4a48bcbdf3687f365f2fb0dc5433f38e203021b3                   17 Jun 20 15:03 UTC
                                                                 dev-3e7414cf7bc5080546e8883826d4c9a38d890ce5                   16 Jun 20 16:50 UTC
                                                                 dev-08fa3d8b68641f941fc7925d3c194e047c8c1bb9                   16 Jun 20 16:36 UTC
                                                                 dev-9a5a828665509d0b14e39629a8ff173b1bdf7470                   02 Jun 20 17:55 UTC
                                                                 prod-efb1dc35d18131d35415d4d884fed7da18d9f569                  02 Jun 20 17:55 UTC

[stefansedich]

I beleive the issue might be is that you have:

annotations:
  fluxcd.io/automated: true

Where you want to have (note the "true"):

annotations:
  fluxcd.io/automated: "true"

The annotation actually gets silently dropped which is always fun 😄, try that out and see if you have any more luck! at first glance the rest of the config looks fine.

[austinbv]

@stefanprodan Thanks for the suggestions sadly still no dice

Before I tried the string here is the output of list-workloads

fluxctl list-workloads -n airflow --k8s-fwd-ns flux
WORKLOAD                                  CONTAINER          IMAGE                                                                                                                   RELEASE   POLICY
airflow:deployment/airflow-flower         airflow-flower     *****.dkr.ecr.*****.amazonaws.com/*****:airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604  ready
airflow:deployment/airflow-scheduler      git-sync           *****.dkr.ecr.*****.amazonaws.com/*****:git-41d01deba5c2a08edd5e006fddd8fe8497539a73          ready
                                          airflow-scheduler  *****.dkr.ecr.*****.amazonaws.com/*****:airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604
                                          git-clone          *****.dkr.ecr.*****.amazonaws.com/*****:git-41d01deba5c2a08edd5e006fddd8fe8497539a73
airflow:deployment/airflow-web            git-sync           *****.dkr.ecr.*****.amazonaws.com/*****:git-41d01deba5c2a08edd5e006fddd8fe8497539a73          ready
                                          airflow-web        *****.dkr.ecr.*****.amazonaws.com/*****:airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604
                                          git-clone          *****.dkr.ecr.*****.amazonaws.com/*****:git-41d01deba5c2a08edd5e006fddd8fe8497539a73
airflow:helmrelease/airflow               airflow            *****.dkr.ecr.*****.amazonaws.com/*****:airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604  deployed                           automated
airflow:statefulset/airflow-redis-master  airflow-redis      docker.io/bitnami/redis:5.0.7-debian-10-r32                                                                             ready
airflow:statefulset/airflow-worker        git-sync           *****.dkr.ecr.*****.amazonaws.com/*****:git-41d01deba5c2a08edd5e006fddd8fe8497539a73          ready
                                          airflow-worker     *****.dkr.ecr.*****.amazonaws.com/*****:airflow-dev-63b3d5ed649cb3e41611c8fc5eb8dfc583433604
                                          git-clone          *****.dkr.ecr.*****.amazonaws.com/*****:git-41d01deba5c2a08edd5e006fddd8fe8497539a73

Interesting it worked! That's crazy

[stefansedich]

@austinbv IIRC you said this issue was fixed now? can we close this issue off?

[austinbv]

@stefansedich I created a PR that I would love to get merged and will close this

[stefansedich]

@austinbv considering the issue was related to flux and the PR is over in the flux repo, makes sense to me to close off the issue over here and track it over in the flux repo.