Feature request: ability to give s3 uris as inputs

[sarahthompson-dev]

Describe the problem

We have our assets already uploaded to other S3 buckets. We want to use these files as inputs to Deadline Jobs without needing to download them and submit them physically with the job bundle. Our current workaround is to add an import step that uses s3 copy to copy the files into the Deadline Worker.

Proposed Solution

It would be nice if we could just give s3 uris as inputs and then be able to work with them within Deadline Cloud as though they had been physical files that were submitted with the job bundle. Even if we needed to provide some sort of mapping yaml or json, that would be no problem.

Example Use Cases

maybe we could have some sort of parameterDefinition for this, similar to PATHs:

 - name: s3UriToAssetDir
    type: S3URI
    objectType: DIRECTORY

 - name: s3UriToAsset
    type: S3URI
    objectType: FILE

[mwiebe]

I've been considering how we might generalize this, and here's an early sketch of some ideas that could work:

1. Extend OpenJD so that PATH parameters can accept URIs in general in addition to local file system paths.
   1. It would be up to the system scheduling the job template to decide which types of URIs to accept.
   2. This would affect path mapping. Currently path mapping maps between different local file system paths. It would now map an s3://[BUCKET]/prefix to a local file system path for the job to use. E.g. in template substitution, {{Param.InputDir}} would be /local/path and {{RawParam.InputDir}} would be s3://...
2. Extend job attachments so that it handles IN/OUT/INOUT data flow appropriately for s3:// and https:// cases.
3. Extend the OpenJD CLI tool to support https:// inputs and s3:// inputs/outputs when boto3 is available.

[sarahthompson-dev]

On first thought, option 2 sounds nice because I like the idea of being able to accept a variety of paths and Deadline Cloud just knows how to handle it. All 3 options sound great though.

[Mathieson]

I like the idea of extending PATH to support URIs because it enables better local dev workflow with openjd run. When in a local dev context, you can pass in a local file instead of needing it to download from the s3 bucket each time.