Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛[BUG]: earth2studio.data.IFS HTTPError from azure #168

Closed
luke-conibear opened this issue Dec 19, 2024 · 1 comment
Closed

🐛[BUG]: earth2studio.data.IFS HTTPError from azure #168

luke-conibear opened this issue Dec 19, 2024 · 1 comment
Assignees
@NickGeneva @pzharrington
Labels
bug Something isn't working

Comments

@luke-conibear
Copy link
Contributor

luke-conibear commented Dec 19, 2024 •
edited
Loading

Version

0.4.0

On which installation method(s) does this occur?

Pip

Describe the issue

earth2studio.data.IFS uses azure to download data.

This can hit HTTPError: 409 Client Error: Public access is not permitted on this storage account.

from datetime import datetime
from earth2studio.data import IFS
from earth2studio.models.px.sfno import VARIABLES
ds = IFS()
time = datetime(2024, 12, 17)
da = ds(time=time, variable=VARIABLES)
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[5], [line 15](vscode-notebook-cell:?execution_count=5&line=15)
     [12](vscode-notebook-cell:?execution_count=5&line=12) ds.client = opendata.Client(source="azure")
     [13](vscode-notebook-cell:?execution_count=5&line=13) time = datetime(2024, 12, 17)
---> [15](vscode-notebook-cell:?execution_count=5&line=15) da = ds(time=time, variable=VARIABLES)
     [17](vscode-notebook-cell:?execution_count=5&line=17) local_file_path_input_data = "/home/azureuser/wxapps/apps/corrdiff_gefs_pre/fcn_inputs.npy"
     [18](vscode-notebook-cell:?execution_count=5&line=18) if Path(local_file_path_input_data).exists():

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:111, in IFS.__call__(self, time, variable)
    [109](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:109) data_arrays = []
    [110](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:110) for t0 in time:
--> [111](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:111)     data_array = self.fetch_ifs_dataarray(t0, variable)
    [112](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:112)     data_arrays.append(data_array)
    [114](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:114) # Delete cache if needed

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:168, in IFS.fetch_ifs_dataarray(self, time, variables)
    [165](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:165) variable, levtype, level = ifs_name.split("::")
    [167](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:167) logger.debug(f"Fetching IFS grib file for variable: {variable} at {time}")
--> [168](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:168) grib_file = self._download_ifs_grib_cached(variable, levtype, level, time)
    [169](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:169) # Open into xarray data-array
    [170](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:170) # Provided [-180, 180], roll to [0, 360]
    [171](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:171) da = xr.open_dataarray(
    [172](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:172)     grib_file, engine="cfgrib", backend_kwargs={"indexpath": ""}
    [173](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:173) ).roll(longitude=-len(self.IFS_LON) // 2, roll_coords=True)

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:228, in IFS._download_ifs_grib_cached(self, variable, levtype, level, time)
    [226](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:226)         request["levelist"] = level
    [227](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:227)     # Download
--> [228](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:228)     self.client.retrieve(**request)
    [230](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/earth2studio/data/ifs.py:230) return cache_path

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:146, in Client.retrieve(self, request, target, **kwargs)
    [145](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:145) def retrieve(self, request=None, target=None, **kwargs):
--> [146](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:146)     result = self._get_urls(request, target=target, use_index=True, **kwargs)
    [147](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:147)     result.size = download(
    [148](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:148)         result.urls,
    [149](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:149)         target=result.target,
    [150](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:150)         verify=self.verify,
    [151](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:151)         session=self.session,
    [152](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:152)     )
    [153](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:153)     return result

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:241, in Client._get_urls(self, request, use_index, target, **kwargs)
    [238](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:238)         seen.add(url)
    [240](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:240) if for_index and use_index:
--> [241](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:241)     data_urls = self.get_parts(data_urls, for_index)
    [243](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:243) return Result(
    [244](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:244)     urls=data_urls,
    [245](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:245)     target=target,
   (...)
    [248](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:248)     for_index=for_index,
    [249](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:249) )

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:262, in Client.get_parts(self, data_urls, for_index)
    [260](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:260) index_url = f"{base}.index"
    [261](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:261) r = robust(self.session.get)(index_url, verify=self.verify)
--> [262](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:262) r.raise_for_status()
    [264](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:264) parts = []
    [265](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/ecmwf/opendata/client.py:265) for line in r.iter_lines():

File /anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/requests/models.py:1024, in Response.raise_for_status(self)
   [1019](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/requests/models.py:1019)     http_error_msg = (
   [1020](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/requests/models.py:1020)         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   [1021](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/requests/models.py:1021)     )
   [1023](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/requests/models.py:1023) if http_error_msg:
-> [1024](https://vscode-remote+amlext-002b-002bsubscriptions-002b4beda260-002d5328-002d4fca-002d9e3e-002d538f8e3f5b05-002bresourcegroups-002bwxapps-002dazureml-002dup-002bproviders-002bmicrosoft-002emachinelearningservices-002bworkspaces-002bazureml-002dhydronn-002bcomputes-002bluke-002dt4-002d4core-002d28gb-003apublic-003aead7805b-002d3ab5-002d4c45-002dbc8b-002db2c957dff611.vscode-resource.vscode-cdn.net/anaconda/envs/azureml_py310_sdkv2/lib/python3.10/site-packages/requests/models.py:1024)     raise HTTPError(http_error_msg, response=self)

HTTPError: 409 Client Error: Public access is not permitted on this storage account. for url: https://ai4edataeuwest.blob.core.windows.net/ecmwf/20241217/00z/ifs/0p25/oper/20241217000000-0h-oper-fc.index

A temporary workaround is to use aws instead.
A longer-term solution might be to make the source an argument and default it to aws. As in #169

from datetime import datetime
from earth2studio.data import IFS
from earth2studio.models.px.sfno import VARIABLES
import ecmwf.opendata as opendata
ds = IFS()
ds.client = opendata.Client(source="aws")
time = datetime(2024, 12, 17)
da = ds(time=time, variable=VARIABLES)
@luke-conibear luke-conibear added ? - Needs Triage Need team to review and classify bug Something isn't working labels Dec 19, 2024
@luke-conibear luke-conibear mentioned this issue Dec 19, 2024
Merged
6 tasks
@NickGeneva NickGeneva removed the ? - Needs Triage Need team to review and classify label Dec 20, 2024
@NickGeneva
Copy link
Collaborator

Fixed in #169

@spirillen spirillen mentioned this issue Jan 31, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NickGeneva NickGeneva

@pzharrington pzharrington

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@NickGeneva @pzharrington @luke-conibear