Non-deterministic evaluation of task's 'when' condition based on output parameters

[Petrax]

My WorkflowTemplate is using a DAG definition with multiple tasks. First task in the DAG is supposed to run some pre-processing and exposes few output parameters that point to a shared filesystem path:

  templates:
  - name: my-pipeline
    ...
    dag:
      tasks:
      - name: start
        template: start-script
        arguments:
          parameters:
          - name: task_name
            value: start
    ...
  - name: start-script
  ...
    outputs:
      parameters:
      - name: param1
        valueFrom:
          path: '{{`{{inputs.parameters.base_dir}}`}}/param1.txt'
          default: ''
      - name: param2
        valueFrom:
          path: '{{`{{inputs.parameters.base_dir}}`}}/param2.txt'
          default: ''
    ...

Subsequent task in the DAG is supposed to run conditionally, based on the outputs of the first task:

  templates:
    dag:
      tasks:
      - name: conditionalTask
        template: conditionalTask
        depends: start
        when: '''{{`{{tasks.start.outputs.parameters.param1}}`}}'' !=
          '''' && ''{{`{{tasks.start.outputs.parameters.param2}}`}}'' !=
          '''''
  ...

In most cases this works fine and the when condition is evaluated correctly based on proper values of output parameters provided by the first task but in some cases the condition evaluates to true even though the file paths used to define the value of these parameters are not pointing to an existing file. This has been observed occurring mostly when there is quite a huge load on the Argo controller due to large number of workflows starting concurrently.
To me, it looks like there is some kind of race condition and the parameter evaluation might not be using the default values of output parameters in case the valueFrom evaluation fails.

Observed with Argo Workflows controller v3.5.11.

Any hints on how to further trace it down or resolve the problem would be very appreciated.