Workflow job with needs output from matrix?

[pluma]

I have a workflow building several docker images and only publishing those images if all builds succeed.

The obvious way to solve this was to have a build job with a matrix of the image names and then a publish job with the same matrix that “needs” the build job. This works fine.

However now I also want to generate outputs during build for each image that I want to re-use during the publish job for the same image. I know that I can work around this using artefacts but if these weren’t matrix jobs I could just do it using outputs.

Is there a special syntax to do access outputs from a matrix job or are matrix jobs simply incapabl of generating outputs? I can’t find any explanation of this in the docs and it doesn’t seem like the outputs could be named dynamically since they need to be explicitly defined at the job level.

[Simran-B]

Hey Alan, matrix jobs can set outputs but they will overwrite each other. The last one wins. As a workaround, you can add a unique parameter for each matrix job and use that to set outputs with different names: [BUG] Jobs output should return a list for a matrix job - #15 by Simran-B

[dhirschfeld]

That link is broken?

[nthienan]

matrix jobs can set outputs but they will overwrite each other

facing this problem now. @Simran-B can you share workaround again?

[pluma]

Hey thanks Simran, looks like I’ll have to duplicate the matrix list for the outputs then. That solution seems obvious in hindsight but didn’t occur to me.

[ghost]

Simran-B:

…matrix jobs can set outputs but they will overwrite each other.

This is not great. I hope the GitHub team will fix this.

[acuD1]

Been trying some workarounds for days now, and no it is no possible. The only think I can think of is to set a secret to a repository and check the value afterward.

So you can't use job level env, you can't set output dynamicaly, thoses values are overwritten by each matrix jobs...

[petersononyeukwu-asurion]

I'm considering storing data in the git repo directly. This post from Simon Willison sparked this idea: https://simonwillison.net/2020/Oct/9/git-scraping/

[atemate]

One of the options is to pass outputs via output artifacts, using GH actions upload-artifact and download-artifact:

name: "CI/CD"

on:
  push:
    branches: [master]

defaults:
  run:
    shell: bash

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        kv:
          - key: one
            val: ONE
          - key: two
            val: TWO
          - key: three
            val: THREE

    steps:
      - id: one
        run: |
          mkdir -p pipelines
          echo ${{ matrix.kv.val }} >> outputs/${{ matrix.kv.key }}.txt

      - uses: actions/upload-artifact@v3
        with:
          name: outputs
          path: outputs/*.txt


  getval:
    runs-on: ubuntu-latest
    needs: [test]
    steps:
      - uses: actions/checkout@v3
      - name: Load outputs
        uses: actions/download-artifact@v3
        with:
          name: outputs
          path: outputs

      - run: |
          ls outputs
          cat outputs/*.txt

[nomeata]

If someone else, like me, is surprised this works, this is the relevant bit from the documentation at https://github.com/actions/upload-artifact:

Each artifact behaves as a file share. Uploading to the same artifact multiple times in the same workflow can overwrite and append already uploaded files:

[outlying]

Right, but before someone will become a bit too enthusiastic by this comment here, it is also good to include follow-up to that quote, from the same source:

Warning: Be careful when uploading to the same artifact via multiple jobs as artifacts may become corrupted. When uploading a file with an identical name and path in multiple jobs, uploads may fail with 503 errors due to conflicting uploads happening at the same time. Ensure uploads to identical locations to not interfere with each other.

[atemate]

Yes, good point: you need to make sure to upload under different names.

[p5]

This no longer works with the latest version of the upload-artifact unfortunately.

Due to how Artifacts are created in this new version, it is no longer possible to upload to the same named Artifact multiple times. You must either split the uploads into multiple Artifacts with different names, or only upload once. Otherwise you will encounter an error.

[gileswells]

@p5 I'm literally staring at this exact line of documentation today and trying to find a solution. I wish that we could bring all the filesystems systems from each matrix branch back down into a single filesystem.

[boozook]

That works because outputs lazily sets to the context. That means orphan fields (with empty value or value that can equals to false) will no assigned to the ctx. So from each parallel job we're writing various keys into one context.

name: Works
on:
  push:
    branches: ["**"]

jobs:
  parallel:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: true
      matrix:
        option:
          - a
          - b

    steps:
      - name: output
        id: output
        run: |
          echo "${{ matrix.option }}=${{ matrix.option }}" >> $GITHUB_OUTPUT

    outputs:
      a: ${{ steps.output.outputs.a }}
      b: ${{ steps.output.outputs.b }}

  finally:
    needs: parallel
    runs-on: ubuntu-latest

    strategy:
      matrix:
        option: ${{ needs.parallel.outputs.* }}

    steps:
      - name: checkout
        uses: actions/checkout@v3

      - name: debug
        run: echo '${{ toJSON(needs.parallel.outputs) }}'

[atemate]

neat, but this solution has a limitation that the output keys must be defined statically in yaml, so for variable set of matrix jobs it won't work :(

[nthienan]

Facing this problem too. My workflow is dynamic matrix jobs so that's not possible to pre-define outputs statically

[crivetechie]

I have used this action to solve the problem, works well with dynamic matrix jobs: https://github.com/marketplace/actions/matrix-outputs-write

[neilime]

We've made a couple of actions to deal with this issue:

• https://github.com/hoverkraft-tech/ci-github-common/blob/main/actions/set-matrix-output/README.md
• https://github.com/hoverkraft-tech/ci-github-common/blob/main/actions/get-matrix-outputs/README.md

[neilime]

@andy-maier it should be fixed with version https://github.com/hoverkraft-tech/ci-github-common/releases/tag/0.8.1

Please let me know

[oerp-odoo]

How is uniqueness guaranteed with that action? I assume it will keep it within github worklow scope? So if someone else starts same workflow, it would not clash?

Let say I have matrix of a, b, c. And I set some keys/values like a-something=10, b-something-=20, c-something=30. So if other workflow would run and use/set same keys (but maybe different values), it would still not clash, right?

[neilime]

@oerp-odoo The GitHub Actions workflow ensures artifact uniqueness by concatenating the workflow's run_id, run_number, and a user-provided artifact-name. Users can enhance this uniqueness by supplying a distinct artifact-name for each run, further ensuring that each artifact is uniquely identified across all runs and workflows. This strategy effectively prevents any clashes or overwrites, maintaining the integrity and traceability of artifacts, especially when multiple workflows are executed concurrently or under similar configurations.

https://github.com/hoverkraft-tech/ci-github-common/blob/main/actions/set-matrix-output/README.md (doc updated)

[oerp-odoo]

@neilime thanks. One more question. How long those artifacts persist? I assume if workflow ends, artifacts are gone? I would not want them to persist longer than it needs to. And I probably can't rely on delete after fetching artifact option, because I might need to fetch it multiple times.

[neilime]

If you are using the action https://github.com/hoverkraft-tech/ci-github-common/tree/main/actions/get-matrix-outputs

There is an input to delete or not he artifact after reading

If you want to delete the artifact on a specific case you can set the remove-artifact input to false and then remove it later in your workflow when you want (using geekyeggo/delete-artifact action by example)

- id: create-artifact
  uses: hoverkraft-tech/ci-github-common/actions/[email protected]
  with:
    value: "test value"
    artifact-name: "test"

- uses: hoverkraft-tech/ci-github-common/actions/[email protected]
  with:
    artifact-name: "test"
    remove-artifact: false

# Delete artifact
- uses: geekyeggo/delete-artifact@v2
  with:
    name: ${{ steps.create-artifact.outputs.artifact-name }}

[omus]

I've developed an action to workaround this problem. It only requires adding one additional step to your job matrix and setting and output: https://github.com/beacon-biosignals/matrix-output