Ease optional access with scatter-like block

[cjllanwarne]

Port of Cromwell issue broadinstitute/cromwell#1852

Equivalent to scatter blocks for arrays, I propose a new ifDefined block for optionals:

workflow foo {
  String? bar
  ifDefined (concreteBar in bar) {
    call baz { input: x = concreteBar }
  }
  output {
    # Just like outputs from inside a scatter become array outputs, isDefined creates optional outputs:
    String? baz_out = baz.out
  }
}

task baz {
  # Non-optional input:
  String x
  # ...
}

[orodeh]

I can see the issue, but I can't say I'm crazy about the syntax. Also, I assume the String in isDefined is superfluous.

ifDefined (concreteBar in bar)

[geoffjentry]

@cjllanwarne Could you provide context on use cases and such? In general it'd be helpful for proposed language constructs to come attached with some concrete examples of "why"

[cjllanwarne]

Sure!

The idea here is to provide syntactic sugar for the boilerplate in the following scenario:

workflow foo {
  [...]
  Int? x = ...
  if(defined(x)) {
    Int real_x = select_first([x])
    call bar { input: x = real_x }
  }
}

task bar {
  Int x
  [...]
}

This construction is becoming more and more common in WDLs as people start using if {} blocks more and getting more and more optional values, and at the same time, Cromwell 30 fixed a bug in WDL parsing and now enforces that optional values cannot be passed into tasks which expect a non-optional input.

[cjllanwarne]

In my opinion the Int real_x = select_first([x]) is an awkward and hard to read line that presents a barrier to entry for WDL readers. The more we can let authors write what they want without having to jump through awkward hoops, the better.

[patmagee]

Non Deterministic workflows (although already supported) is one thing that I always struggle with, as it defeats the underlying purpose of reproducibility. I always wonder If this type of construct is occurring more and more, whether its something that we should encourage, or try to steer users away from.

That being said if this is something everyone wants, then I will entertain the idea.
When it comes to the specific proposed solution I am with @orodeh on this. I am not really a huge fan of the syntax. the in keyterm is more strongly associated with iterable types and it a bit confusing.

I would not add any additional constructs or execution blocks. I believe this can be solved by properly employing current constructs then adding a better select engine function.

The select function would take any optional element, and return the concrete version of the underlying element. However, if the optional value is not defined, then the select function would throw an error and that stage of the workflow would fail. Yes this introduces a point of failure if used improperly, however I think it is a reasonable burden on the user to ensure that they use the select function within an if block.

workflow foo {
  [...]
  Int? x = ...
  if(defined(x)) {
    call bar { input: x = select(x) }
  }
}

task bar {
  Int x
  [...]
}