decide on attributes and metadata for downscaled data

[dgergel]

We need to iterate and decide on how we want to set up our attributes and metadata for downscaled zarr stores - this is just a placeholder to start the discussion.

[brews]

I think there is a more essential low-hanging fruit version of this. As it is right now, we're just dumping blob data into azure storage. We need at least enough basic metadata to know what we're looking at, at the simplest level (source_id, variable_id, etc...), even if that information is not encoded in the data path.

[dgergel]

That makes sense. So first pass can be all of the basic identifiers - e.g. SSP, model, variable, frequency, etc - along with methods used (e.g. QDM/AIQPD, wet day frequency for precip). We can iron out the CF-conventions-compliance part of this as a next step. Sound good?

In terms of what data actually needs metadata, I'm thinking the bias corrected and downscaled outputs - any intermediate outputs as well? Cleaned and rechunked CMIP6 input data to bias correction perhaps?

Thinking we add a function in dodola for now that adds attributes based on the step of the workflow and on the tuple identifiers unique to each GCM/variable/scenario. We would have those as parameters in the workflow already. How does that sound to you @brews?

[brews]

@dgergel

Thinking we add a function in dodola for now that adds attributes based on the step of the workflow and on the tuple identifiers unique to each GCM/variable/scenario. We would have those as parameters in the workflow already. How does that sound to you @brews?

The immediate issue is that metadata needs to be preserved between workflow steps because it's useful to validate and write final-ish output. All of this "immediately essential" data (stuff we use for I/O) is present as soon as we download it from CMIP6-in-the-cloud. We just need to hold on to it. Right now, ClimateImpactLab/dodola#116 is likely the most obvious offender.

[brews]

For an example of what we currently have in place, here is a copy of the Dataset metadata for output:

• downscaled
• biascorrected

Our additions have a dc6_ prefix. This separates original metadata from our contributions.

I need additional feedback to continue. Any other input?