🐛[BUG]: 'tp06' - 6 hourly accumulated precipitation cannot be fetched from CDS for FuXi

[purohitkaran20]

Version

v0.4.0

On which installation method(s) does this occur?

Source

Describe the issue

The FuXi model requires the variable named 'tp06' i.e. 6-hourly accumulated total precipitation as input to run. And the API request is trying to fetch this variable directly from the CDS. But this variable is not know to CDS. CDS only knows about 'tp' i.e 1-hourly accumulated total precipitation. I think this is reason I am getting the below error while trying to run the FuXi model.

`
model = FuXi.load_model(FuXi.load_default_package())
ds = CDS()
io = NetCDF4Backend("fuxi.nc")

run(["2024-01-01"], 10, model, ds, io)
`

Error:
`
2025-01-11 00:39:58.006 | INFO | earth2studio.run:deterministic:75 - Running simple workflow!
2025-01-11 00:39:58.006 | INFO | earth2studio.run:deterministic:82 - Inference device: cuda
2025-01-11 00:40:06.377 | ERROR | earth2studio.data.cds:_build_requests:255 - variable id tp06 not found in CDS lexicon

KeyError Traceback (most recent call last)
Cell In[15], line 11
8 ds = CDS()
9 io = NetCDF4Backend("fuxi.nc")
---> 11 run(["2024-01-01"], 10, model, ds, io)

File /scratch/users/karan/egu_25/earth2studio/earth2studio/run.py:96, in deterministic(time, nsteps, prognostic, data, io, output_coords, device)
93 interp_to = None
94 interp_method = "nearest"
---> 96 x, coords = fetch_data(
97 source=data,
98 time=time,
99 variable=prognostic_ic["variable"],
100 lead_time=prognostic_ic["lead_time"],
101 device=device,
102 interp_to=interp_to,
103 interp_method=interp_method,
104 )
106 logger.success(f"Fetched data from {data.class.name}")
107 # sphinx - fetch data end
108
109 # Set up IO backend

File /scratch/users/karan/egu_25/earth2studio/earth2studio/data/utils.py:90, in fetch_data(source, time, variable, lead_time, device, interp_to, interp_method)
88 for lead in lead_time:
89 adjust_times = np.array([t + lead for t in time], dtype="datetime64[ns]")
---> 90 da0 = source(adjust_times, variable) # type: ignore
91 da0 = da0.expand_dims(dim={"lead_time": 1}, axis=1)
92 da0 = da0.assign_coords(lead_time=np.array([lead], dtype="timedelta64[ns]"))

File /scratch/users/karan/egu_25/earth2studio/earth2studio/data/cds.py:132, in CDS.call(self, time, variable)
130 data_arrays = []
131 for t0 in time:
--> 132 data_array = self.fetch_cds_dataarray(t0, variable)
133 data_arrays.append(data_array)
135 # Delete cache if needed

File /scratch/users/karan/egu_25/earth2studio/earth2studio/data/cds.py:164, in CDS.fetch_cds_dataarray(self, time, variables)
162 if isinstance(variables, str):
163 variables = [variables]
--> 164 requests = self._build_requests(time, variables)
165 # pbar = tqdm(
166 # total=len(requests),
167 # desc=f"Fetching CDS for {time}",
(...)
170
171 # Fetch process for getting data off CDS
172 def _fetch_process(request: CDSRequest, rank: int, return_dict: dict) -> None:

File /scratch/users/karan/egu_25/earth2studio/earth2studio/data/cds.py:256, in CDS._build_requests(self, time, variables)
254 except KeyError as e:
255 logger.error(f"variable id {variable} not found in CDS lexicon")
--> 256 raise e
258 dataset_name, cds_variable, level = cds_name.split("::")
259 request_id = f"{dataset_name}-{cds_variable}"

File /scratch/users/karan/egu_25/earth2studio/earth2studio/data/cds.py:253, in CDS._build_requests(self, time, variables)
250 for i, variable in enumerate(variables):
251 # Convert from Nvidia variable ID to CDS id and modifier
252 try:
--> 253 cds_name, modifier = CDSLexicon[variable]
254 except KeyError as e:
255 logger.error(f"variable id {variable} not found in CDS lexicon")

File /scratch/users/karan/egu_25/earth2studio/earth2studio/lexicon/base.py:25, in LexiconType.getitem(cls, val)
23 def getitem(cls, val: str) -> tuple[str, Callable]:
24 """Retrieve variable name."""
---> 25 return cls.get_item(val)

File /scratch/users/karan/egu_25/earth2studio/earth2studio/lexicon/cds.py:131, in CDSLexicon.get_item(cls, val)
128 @classmethod
129 def get_item(cls, val: str) -> tuple[str, Callable]:
130 """Return name in CDS vocabulary."""
--> 131 cds_key = cls.VOCAB[val]
133 def mod(x: np.array) -> np.array:
134 """Modify name (if necessary)."""

KeyError: 'tp06'
`

[NickGeneva]

Hi @purohitkaran20

Thanks for opening an issue, this is an expected error. You are correct, we do have tp06 in the CDS datasource since its not provided.

Two options you can consider:

1. Use a data source with tp06 like Weather Bench 2. Refer to the lexicon files to see if tp06 is known (or can always test manually).
2. Create a custom data source that either calculates tp06 or approximate it.

Thanks!

[purohitkaran20]

Thanks, @NickGeneva, for clarifying.