Enabling CRAM downloads for RD collaborators

[cassimons]

We need a process by which a collaborator can gain access to the CPG-generated CRAM files for the samples they have provided CPG. As a principle, it is important we can provide collaborators access to their data as required. However, this must be provided in a way that is:

• Secure.
• Ensures egress costs are appropriately budgeted for.
• Mitigates the risk of unexpected egress cost blowouts for either CPG or the collaborator.
• Has low (no?) per dataset requirements for CPG staff to enable (This should be considered a regular expected ad hoc event, not an exception).

A fully-featured implementation to enable this would require a firm policy to be developed informing who is responsible for costs associated for what specific data from what projects (development of this is sitting with the PM team). This ticket aims to find the technical solution/s that we can use to support that policy.

In the short term, we have current requests for access to limited sets of CRAMs from two RD projects. In both cases, CPG has agreed to cover the egress costs for these specific downloads and the collaborators are currently compiling lists of the specific CRAMs required. Once the lists are available it would be good to use them as a pilot to establish an interim system for enabling downloads.

[illusional]

Small update on this issue that this is partially resolved:

• We have a new shared project to which we can control egress costs against (note it does have a 6 hour-ish lag time though, but that's a GCP billing delay).
  • Eg: https://github.com/populationgenomics/analysis-runner/blob/3c928e64508b64bdff20ae9016be74a5b09a7ddf/stack/Pulumi.mito-disease.yaml#L6
• It's mild effort to copy crams for now to a release bucket (until we can enable requester-pays on all main buckets)
• It supports ability to use a service-account credentials to interact directly with GCS, or it's possible to generate a set of signed-URLs for files based on that service-account (so egress costs are still protected).