en/latest/configure-archive-logs/

[giscus[bot]]

en/latest/configure-archive-logs/

https://argo-workflows.readthedocs.io/en/latest/configure-archive-logs/

[gyanprakash48]

I set artifact repository as below in workflow-controller-configmap, but when trying to run a workflow with archiveLogs: true, it's throwing error You need to configure artifact storage. More information on how to do this can be found in the docs: https://argoproj.github.io/argo-workflows/configure-artifact-repository/. Also tried with creating a config artifact-repositories. Any suggestion what else i need to do ?

artifactRepository: |
    azure:
      endpoint: https://argodat3o77q.blob.core.windows.net
      container: artifacts
      blob: {{workflow.name}}/{{pod.name}}
      accountKeySecret:
        name: argo-artifcats-storage-credentials
        key: account-access-key

[brianloss]

Sure, you're welcome!

[monaka]

As you might know, AAD Pod Identity is at sunset. You can use Azure Workload Identity instead.
I checked and confirmed workflow-server (v3.4.11) works with Workload Identity.

Additional code to values.yaml is like this:

server:
  podLabels:
    azure.workload.identity/use: "true"
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: <<your client-id>>

[Borrelworst]

Hi Monaka,

I am trying to get it working in a slightly different way, but so far so good. What I have done:

• Created the service account
• Created the federation identity
• It is all linked together

If I try to manually query the storage account from within the pod it works fine. However, if I run the workflow I get the following error:

Error (exit code 1): unable to upload file /tmp/argo/outputs/logs/main.log to Azure: ===== INTERNAL ERROR ===== DefaultAzureCredential: failed to acquire a token. Attempted credentials: EnvironmentCredential: incomplete environment variable configuration. Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set ManagedIdentityCredential: IMDS token request timed out AzureCLICredential: fork/exec /bin/sh: no such file or directory

In the pod, this works fine:

:$ which /bin/sh
/bin/sh

So what is missing?

[Borrelworst]

After some debugging, it looks like using workload-identity does not work when AAD Pod Identity is installed as well. When AAD Pod Identity is installed, it takes precedence of workload-identity, causing it to fail. Is there an option to specify what identity type needs to be used ?

[monaka]

Workload Identity is a new replacement for AAD Pod Identity.
So I suppose there is no way to make them coexist.

AAD Pod Identity is deprecated. https://github.com/Azure/aad-pod-identity#-announcement
And the AKS Managed add-on also will be deprecated in Sept. 2024.

[alejoGT1202]

Hello is there a way I can the path where I would like to save my logs in the s3 bucket?

[Paritosh-Anand]

Yes, there is a way to specify prefix path

keyFormat: prefix/in/bucket     #optional

It is mentioned here as well s3-compatible-artifact-repository-bucket-such-as-aws-gcs-minio-and-alibaba-cloud-oss

[skight-shen]

I've configured with mysql as the manul. Tables are created in the databases as metioned in the manul. However, it didn't archive the workflow. I'can see any workflow in UI and in the database. Can anyone help? The workflow run in a customized namesapce which is different from the default "argo". Any suggestions?

[skight-shen]

achive: true has setted in the config.

[skight-shen]

fixed. It mismatched the labels.

[nicolas-vivot]

@skight-shen
Can you please give some details about what was wrong and what you have fixed ?
I am in the same situation, did not have a deep look at it yet.

[nicolas-vivot]

Got it. There is a default configuration to archive workflows matching label selector. Did not pay attention too.

[skight-shen]

It seems to be not stable. Have you met this issue?

[ericsgagnon]

Any way to change postgres schema? We're a small group and I prefer to colocate these types of logs. The ability to use a schema like argo would be great.

[nirdz]

I agree with the question. It seems that Argo creates the necessary tables under the default "public" scheme and I have not found a way to configure the schema.

[panicboat]

Hello.

I would like to see archived workflow logs in the GUI.

When I set it up as following, I get an error with the message step group deemed errored due to child steps[0].hello1 error: failed to resolve {{steps.name}} and cannot start the workflow.
For dag, the error message is task 'artifact-repository-s3-workflow.start' errored: failed to resolve {{tasks.name}}.

Is there a recommended keyFormat for storing multi-step and DAG logs?

MultiStep

data:
  multi-step-repository: |
    s3:
      endpoint: s3.amazonaws.com
      bucket: xxxx-xxxx-workflow-artifacts
      region: ap-northeast-1
      keyFormat: {{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{workflow.name}}/{{steps.name}}/{{workflow.creationTimestamp}}

DAG

  dag-repository: |
    s3:
      endpoint: s3.amazonaws.com
      bucket: xxxx-xxxx-workflow-artifacts
      region: ap-northeast-1
      keyFormat: {{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{workflow.name}}/{{tasks.name}}/{{workflow.creationTimestamp}}

[panicboat]

I found that by not specifying keyFormat, logs are saved for each Pod.
Solved the problem for now.

[Borrelworst]

I am posting as I am not sure how to configure it according to my needs:

I try to set it up to work with AzureBlob. I have multiple namespaces, all with their own workflows. When I configure this directly in the workflow controller configmap, I just get a static folder and all workflows try to write to that single file it creates (ex. dev/cluster/main.log). Then the archive logs just work for the first workflow that ran, but for subsequent workflows I get the following error in the logs:
"Artifact Server returned internal error" error="artifact not found: main-logs, isInput=false, Workflow....

What I would like is that every workflow put it logs in a separate folder (ex. dev/cluster/workflow1/main.log, dev/cluster/workflow2/main.log)

I am just not sure if that is even possible when configuring it only in the workflow controller configmap: I would think every workflow need to have it's own reference for where to write the logs. Does anyone has an idea how to configure this, and how does this work for s3 for example?

Thanks!

[Borrelworst]

It looks like I do get it working, after some restarts and retries I can just reference it from a workflow directly (and not configure it in the workflow controller configmap).

[Borrelworst]

One last question: If I want to get this working with Azure Blob in combination with the useSDKCreds: true and a managed idenity, how can I get that working? The moment I add both AZURE_TENANT_ID and AZURE_CLIENT_ID I get the following error:

Only AZURE_TENANT_ID and AZURE_CLIENT_ID are set ManagedIdentityCredential: IMDS token request timed out AzureCLICredential: fork/exec /bin/sh: no such file or directory

If I don't add anything, the error is:

missing environment variable AZURE_TENANT_ID

I think it should work, but I am missing some documentation or an example on how to configure this correctly.

[tooptoop4]

@Borrelworst see #13293 (comment)

[sfc-gh-rpeng]

Hi Argo experts,

Is there a way to mark logs artifacts are optional so that the workflow will not fail when logs archival fails?
For other artifacts this is achieved by the optional field here.

[koder11111]

Hello, Is it possible to save the wait.log file in addition to the main.log file?

[agilgur5]

A feature request was opened for this: #12640