CF variable group not found in netCDF4 file; also not being able to load cube should tell the user why #3544
Comments
|
UPDATE I changed the |
|
So this is how a is this happening because iris still works with CF=1.5 maybe? |
|
@valeriupredoi Good to hear from you 😃 Looking at the Would it be possible for you to share the actual NetCDF file with me? Or explain how and where I can download it from? That's the only way that I'm going to easily progress this for you. Thanks! |
|
Hi Bill, On JASMIN, I'm finding that this problem occurs will all the files that I have looked at matching: ls /badc/cmip6/data/CMIP6/CMIP///historical/r*/O*/volcello/gr/latest/volcello*: If you can't access jasmin, these files are also available on ESGF. MIP era: |
|
@bjlittle extra extra 😁 Man, I tried |
|
@bjlittle here's a clue I got from and indeed asking for a type it looks like the fill value is both |
|
where |
|
What I don't understand is why this would happen for so many different models? Is there an issue in the CMOR table? |
|
@valeriupredoi Thanks. I'm offline pretty much today (only capable of servicing GitHub), but I'll spin up on this early tomorrow morning and investigate further. |
|
@ledm What ever it is, it seems to be systemic. |
|
Thanks @bjlittle, any help you can give us on this tomorrow would be great. It's a fairly urgent problem for us. We'll keep looking at it and post anything we find here - but we understand that you might be see anything until tomorrow. Cheers! |
|
@ledm and @valeriupredoi Of course, this might be symptomatic of a dependency issue, rather than a data issue... just guessing as I have no laptop, only my mobile. What version of |
|
Might be relevant, might absolutely not... |
|
|
|
will attempt installing |
|
Oh but noooo - it is python3.8 and we can't move to 3.8 just yet, I'll attempt anyways just to check the pure iris functionality |
|
I don't think you can move to |
|
Also The latest available on |
still no luck! BTW iris and py3.8 installed OK in the same env, and the import works fine. Can't upgrade to 4.7.2, the env refuses to solve at the moment |
|
reverted to older versions: still no joy! It doesn't seem to be a dependency issue 😿 |
|
Interesting...well if it's a data problem, then that might be potentially huge. Hope that's not the case... |
|
How recent is the CMIP6 data? |
|
the files that @ledm has listed have been uloaded either on Jun 11, Aug 2 or Nov 16 |
|
Does all |
|
I found it!! So the actual data variables and promoted variables for so those are empty - and this is because you need to promote it to real var, man 🍺 |
|
@valeriupredoi Could you please mind your use of language, this is a public forum.
So I'll have a think about all this, and whether there is a workaround or fix that might assist here. The fact that CMIP6 is now producing self-referencing metadata is questionable, but that's not going to make its data go out of distribution any time soon, or ever, for the community. |
sorry, edited it 😁
Indeed, maybe create a list of variables that can be both ways? I think I have this in but it's not working -> any idea how to include |
|
Two quick comments:
Never say never. The analogous entry in
|
|
@bjlittle so I managed to add the MeasureVariable as DataVariable to the but the final |
Is it worth raising this as an issue with the CMOR guys? This is the CMOR table for |
|
It's perhaps more an issue of the data request, but yeah, definitely worth raising there. |
|
Just checked and this issue has been already been fixed in If you have an environment with all the installed dependencies of |
|
I'm now happy to close this issue, if you guys are... |
|
Before we do that, @valeriupredoi, how do we implement this fix in ESMValTool? |
|
Actually, lets continue that discussion on the ESMValTool github: |
|
Cheers much Bill!! Lee, we can't implement it in an automated way just yet,
since the iris conda package is not yet to the master dev version, but I
guess we can grab the branch as Bill says and install from source, lemme
try do it myself and will tell you how you can do it too (I never installed
iris from source before)
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
…On Wed, 20 Nov 2019, 10:52 Lee de Mora, ***@***.***> wrote:
Before we do that, @valeriupredoi <https://github.com/valeriupredoi>, how
do we implement this fix in ESMValTool?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3544?email_source=notifications&email_token=AG5EFIYO2GDB5FSSXQQHWMLQUUJFDA5CNFSM4JOS4562YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEERSAZI#issuecomment-555950181>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AG5EFI4SXJFBOHPMHXGKFPTQUUJFDANCNFSM4JOS456Q>
.
|
@ledm Yeah, that kinda make sense to me. Ideally, you'd just want to Until then, perhaps you're happy to run with something like installing all the I'll keep this issue open for a while in case there is any follow-on chatter. |
|
Cheers, Bill! In theory all deps should have been installed by conda in the
esmvaltool env, unless iris 3 has new deps vs 2.2.1. Ill try as soon as I
get in work. Much thanks to Bill for being so on the ball 🍺
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
…On Wed, 20 Nov 2019, 10:59 Bill Little, ***@***.***> wrote:
Actually, lets continue that discussion on the ESMValTool github:
ESMValGroup/ESMValTool#1442
<https://github.com/ESMValGroup/ESMValTool/issues/1442>
@ledm <https://github.com/ledm> Yeah, that kinda make sense to me.
Ideally, you'd just want to conda install -c conda-forge iris=3.*, but
that's not available just yet, well most likely not until next month.
Until then, perhaps you're happy to run with something like installing all
the iris dependencies and then use something along the lines of pip
install git+https://github.com/scitools/iris/ --no-deps... which should
pull in the master version of iris. Something like that.
I'll keep this issue open for a while in case there is any follow-on
chatter.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3544?email_source=notifications&email_token=AG5EFI2U3HF2CK3FPP4BA3DQUUKCHA5CNFSM4JOS4562YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEERSXHY#issuecomment-555953055>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AG5EFI7H666LI76HZPCSAM3QUUKCHANCNFSM4JOS456Q>
.
|
|
@bjlittle @ledm I have installed iris=3 and it loads the file like a charm 🎉 Absolute ledgs, the iris folk 🍺
NOTE: this will keep the source in the
Loading of volcello files will be biutiful 😁 |
|
@bjlittle go ahead and close this, man. Also, note that the installation via pip is not working due to which is something Ive run into before for other packages and not even related to iris, it's |
|
I wrote the above before V's last comment appeared, but it is confusing for me because I always do my iris source installation with |
I use |
|
So somehow pyke is missing. Hm. |
|
it's not, it's in the conda env: but pip is somehow unable to find it, maybe it needs the pypi version, dunno, I have encountered this before and that's why we stayed with the conda-forge installation for iris and not the pypi one. Altho I seem to recall fixing it. Anyway - outside the scope of this issue - let's wait for the "official" release of iris 3 but it already looks very promising 😁 |
|
...and now 😄 |
|
@valeriupredoi @zklaus Ohh that interesting issues with I can recreate that locally, which is a bit of a 🐷 To be honest, I'm keen to purge our use of In the meantime, I'll close this issue. Thanks guys 👍 🍻 😄 |
|
No, thank you Bill 🍺 The pyke issue is a non issue when installing
from conda so we'll be fine when you guys release the conda package. In the
meantime you'll be happy to hear I produced IPCC AR6 plots with
ESMValTool+iris3 😁
Dr Valeriu Predoi.
Computational scientist
NCAS-CMS
University of Reading
Department of Meteorology
Reading RG6 6BB
United Kingdom
…On Fri, 22 Nov 2019, 09:41 Bill Little, ***@***.***> wrote:
@valeriupredoi <https://github.com/valeriupredoi> @zklaus
<https://github.com/zklaus> Ohh that interesting issues with pyke on
installation...
I can recreate that locally, which is a bit of a 🐷
To be honest, I'm keen to purge our use of pyke from iris... sooner
rather than later would be a very good move indeed. That said, I'll
investigate this side issue of pyke causing install to barf. The pyke
package is needed to build the pyke rules, but you should be able to
build and install iris in a clean way. I think this is fixable, and on us
to do that... so leave that with me.
In the meantime, I'll close this issue.
Thanks guys 👍 🍻 😄
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3544?email_source=notifications&email_token=AG5EFI6HY7OUPXBIZLBNIJLQU6SM5A5CNFSM4JOS4562YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE5DQHI#issuecomment-557463581>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AG5EFI62NDDYGPUHXSJIB6LQU6SM5ANCNFSM4JOS456Q>
.
|
|
Just a post scriptum on the pyke installation error: This happened to me when I set up an environment first with |
|
most interesting @zklaus - I shall proceed to actually try to figure out what's the hap here 🍺 |
Hi guys, I have been asked by @ledm why a certain
NetCDF_v4 (priority 5)file can not be loaded byirisbutncdumpworks fine and in actuality it is a netCDF file with metadata and data in it (I am attaching thencdump -houtput; the results ofload_raworloadare empty lists andload_cubereturnsiris.exceptions.ConstraintMismatchError: no cubes found). It proves out that the reason iris can not load any cube out of the file isiris.fileformats.netcdf.load_cubes()returns no cube since it can not find any CF variable ie the dictcf.cf_group.data_variablesis totally empty. Why this is is beyond my paygrade, my two cents is that the attribute:Conventions = "CF-1.7 CMIP-6.2" ;is wonky. But you know better! Can I suggest one thing and ask anthoer pls:irisactually told the userI can't bloody load the cube because I couldn't find any CF vars;Many thanks in advance! 🍺
@bjlittle @pelson @lbdreyer
problem_file.txt
The text was updated successfully, but these errors were encountered: