Nimrod file format #3647

MoseleyS · 2020-01-27T14:40:35Z

Enriches the meta-data inherited from Nimrod-format data and the data-types associated with coords.
Requires new iris-test-data from SciTools/iris-test-data#55

…t handler.

lib/iris/fileformats/nimrod_load_rules.py

…tions. Adds forecast_period coord. Adds more Nimrod data to unit-tests to cover known operational data.

lib/iris/fileformats/nimrod_load_rules.py

- Improves handling of missing data values. - Improves handling for strings with null characters present. - Replaces instances of "unknown" units with Unit("1").

lib/iris/fileformats/nimrod_load_rules.py

- Adds title and institution attributes. - Excludes some attributes when handling radar data as these have different meanings.

Ran black on the code base and committed the results. I think I've blacked-out.

lib/iris/fileformats/nimrod_load_rules.py

…cted, these test results need updating as they were NOT based on an accurate OSGB grid.

- Distinguishes between precipitation rate and accumulation

- Raises a warning if the coordinate reference system is incomplete

- Expands doc-strings for coord system functions

- Rearranges warning message to make it easier to read.

pp-mo · 2020-05-22T14:20:28Z

Hi @MoseleyS .
Warning : tests have not been run on the latest commit..
Travis testing is not triggering reliably + I'm not sure why.
I see you had this problem yesterday, whereas I was getting tests earlier on today but it just stopped for me.

- Moves origin_corner() to before horizontal_grid()

Adding unit tests for unit function

Removed unnecessary dictionaries from name function

fionaRust

A few more small comments and at least one thing I still don't understand fully.

I think we can also cut back on the number of cml files and/or the number of cubes in each one. A lot of them are testing data with the same kind of data, just with different names. We may want to think about how useful this is. At the moment it is impossible to check it all carefully because there is so much of it.

fionaRust · 2020-05-26T13:32:32Z

lib/iris/fileformats/nimrod_load_rules.py

        )
    elif field.horizontal_grid_type == 1:
        coord_sys = iris.coord_systems.GeogCS(**ellipsoid)
    else:
-        raise TranslationError(


Why was this changed?

I can't clearly see the context, but I think it is because this Translation Error is triggered by another function and doesn't need to be in the code-base twice.

That's fine

fionaRust · 2020-05-26T13:36:17Z

lib/iris/fileformats/nimrod_load_rules.py

+        add_attr(key)
+
+    # Remove member number from cube_source
+    match = re.match(


Might need to explain this to me

Comment expanded to explain what and why is going on here.

fionaRust · 2020-05-26T13:37:37Z

lib/iris/fileformats/nimrod_load_rules.py

@@ -909,6 +926,7 @@ def run(field):
    experiment(cube, field)

    # horizontal grid
+    origin_corner(cube, field)
    horizontal_grid(cube, field)


Should put in horizontal_grid docstring that it should come after origin_corner

fionaRust · 2020-05-26T15:27:50Z

lib/iris/tests/results/nimrod/probability_fields.cml

+      </coord>
+      <coord datadims="[0]">
+        <auxCoord bounds="[[0.0, 304.8],
+		[0.0, 30000.0]]" id="a2cbc1ea" points="[0.0, 0.0]" shape="(2,)" standard_name="height" units="Unit('m')" value_type="float32">


I'm not sure I understand a lot of the probability metadata in this file. Can we go through it together?

fionaRust · 2020-05-26T15:35:02Z

lib/iris/tests/results/nimrod/u1096_ng_bsr05_precip_accum60_2km.cml

@@ -0,0 +1,43 @@
+<?xml version="1.0" ?>


Do we need this test as well as the lib/iris/tests/results/nimrod/u1096_ng_bmr04_precip_2km.cml test

Also would it be possible to rename the test and result files to make them a bit easier to understand.

The file names come from the operational suite file names. Only the initial date string has been stripped off. I did this so I could be confident I'd tested all the use-cases I knew of.

fionaRust · 2020-05-26T15:38:49Z

lib/iris/tests/results/nimrod/u1096_ng_ek00_cloud3d0060_2km.cml

+        <dimCoord id="201cc2ba" points="[1580180400]" shape="(1,)" standard_name="forecast_reference_time" units="Unit('seconds since 1970-01-01 00:00:00', calendar='gregorian')" value_type="int64"/>
+      </coord>
+      <coord datadims="[0]">
+        <dimCoord id="a2cbc1ea" points="[30.0, 60.0, 90.0, 120.0, 150.0, 180.0, 210.0,


Is it expected that this will load two separate cubes? From a first look they look like they should be merge-able?

Yes. This is correct. The difference is <attribute name="num_model_levels"/> where value="57" or value="26". These are two different level-sets that should remain separate.

pp-mo · 2020-05-27T13:59:28Z

Hi @MoseleyS what's the state of this now, are there still points above you want to address ?

I'm currently proposing an addition to the load functions if you wanted to use that, as you suggested above

MoseleyS · 2020-05-28T08:30:01Z

Hi @MoseleyS what's the state of this now, are there still points above you want to address ?

I'm currently proposing an addition to the load functions if you wanted to use that, as you suggested above

Hi Patrick. Yes, there are still a few points to address here. I'm hoping to look at them today (28th). The loader kwargs looks like it will allow us to optionally disable some things - I'll have a look more closely at that.

…mrod_file_format * 'nimrod_file_format' of github.com:MoseleyS/iris: Removed unnecessary dictionaries from name function Adding unit tests for unit function

…cies.

MoseleyS · 2020-05-28T10:42:12Z

Hi @pp-mo
I've added a kwarg to allow the user to suppress the handling of known meta-data issues. I've also added a test for this, but of course the test fails because this PR doesn't include the one that introduces kwargs handling. Not quite suite how to proceed with this now.

pp-mo · 2020-05-28T11:24:47Z

lib/iris/tests/test_nimrod.py

+        ):
+            iris.load(
+                tests.get_data_path(("NIMROD", "uk2km", "cutouts", datafile,)),
+                handle_metadata_errors=False,


For the load api extension proposed in #3720, this isn't the correct syntax.
Instead, it will need to set the loader_kwargs keyword,
e.g. in this case ..., loader_kwargs={'handle_metadata_errors':False})

Of course, #3720 isn't agreed yet,either ...

pp-mo · 2020-05-28T11:29:14Z

of course the test fails because this PR doesn't include the one that introduces kwargs handling

Well, you can either wait, or you could re-organise the test to use the format-specific interface instead.
I.E. replace the existing iris.load call with iris.fileformats.nimrod.load_cubes.
It's not quite the same, as there is no merge step, but in the context of this test I think it should work fine.

- handle_metadata_errors is no longer a kwarg via iris.load_cube as the functionality is not in master yet and wasn't coded correctly here anyway - Removes "_above_threshold" from cloud field id 161 - Removing of zero-length bounds has been added but disabled until the merge_cube method stops stripping bounds where one cube has bounds=None.

fionaRust

I'm going to approve this now. You've addressed my latest set of comments and I've checked that the handle_metadata_errors functionality will turn off the right bits of the code if it is enabled. I've not checked there are sufficient tests for this functionality

pp-mo · 2020-06-04T14:35:20Z

Many thanks for all your patience @MoseleyS !

MoseleyS added 3 commits January 24, 2020 16:21

Attempt to copy in changes developed locally to the nimrod file forma…

edd35fa

…t handler.

Updates unit-test results

c7cac57

Fixes precision of data and coords. Updates existing unit-tests.

d7e6b1d

SciTools-assistant added the Blocked: CLA needed See https://scitools.org.uk. Submit the form at: https://scitools.org.uk/cla/v4/form label Jan 27, 2020

stickler-ci reviewed Jan 27, 2020

View reviewed changes

MoseleyS added 3 commits January 27, 2020 16:33

Updates existing unit-tests.

a428a0b

Removes trailing whitespace.

f16fb3a

Improves accuracy and precision of meta-data handling and known excep…

22f2fcc

…tions. Adds forecast_period coord. Adds more Nimrod data to unit-tests to cover known operational data.

stickler-ci reviewed Jan 30, 2020

View reviewed changes

lib/iris/fileformats/nimrod_load_rules.py Outdated Show resolved Hide resolved

lib/iris/fileformats/nimrod_load_rules.py Outdated Show resolved Hide resolved

Stickler issues solved.

f21d26d

MoseleyS mentioned this pull request Jan 30, 2020

Adds more Nimrod-format data SciTools/iris-test-data#55

Merged

MoseleyS added 2 commits January 30, 2020 15:41

Updates iris-test-data commit reference.

7dd34b2

Adds whats-new entry

96125f4

SciTools-assistant removed the Blocked: CLA needed See https://scitools.org.uk. Submit the form at: https://scitools.org.uk/cla/v4/form label Jan 31, 2020

Improves handling of Nimrod-format files as seen from Radarnet.

62e56b1

- Improves handling of missing data values. - Improves handling for strings with null characters present. - Replaces instances of "unknown" units with Unit("1").

stickler-ci reviewed Feb 17, 2020

View reviewed changes

lib/iris/fileformats/nimrod_load_rules.py Outdated Show resolved Hide resolved

MoseleyS added 4 commits February 17, 2020 14:34

Corrects indentation for Stickler Bot.

8a9c905

Updates handling of specific attributes.

1e9b922

- Adds title and institution attributes. - Excludes some attributes when handling radar data as these have different meanings.

Tries changing the source of the iris-test data.

1e4eb58

Adds decorator to skip data tests when TEST_MINIMAL=True.

28cf783

TomekTrzeciak mentioned this pull request Feb 18, 2020

nimrod monkey patch metoppv/improver#1160

Merged

1 task

MoseleyS added 2 commits February 21, 2020 12:36

Removes unnecessary test.

c97d048

Corrects crs origin information in some old Nimrod files.

ae1d3a1

MoseleyS force-pushed the nimrod_file_format branch from 2b6e7c6 to ae1d3a1 Compare February 21, 2020 15:54

Blacker than the ace of spades.

f7c4ae9

Ran black on the code base and committed the results. I think I've blacked-out.

stickler-ci reviewed Feb 24, 2020

View reviewed changes

lib/iris/fileformats/nimrod_load_rules.py Outdated Show resolved Hide resolved

TomekTrzeciak mentioned this pull request Feb 28, 2020

Port appropriate code to Iris metoppv/improver#468

Open

MoseleyS added 3 commits March 6, 2020 13:38

As the crs origin information in some old Nimrod files has been corre…

ed451e5

…cted, these test results need updating as they were NOT based on an accurate OSGB grid.

Improves regex syntax to make Stickler happy.

f5e22e6

Adds missing blank line for black.

2aca4c5

MoseleyS added 5 commits May 22, 2020 14:48

Added element information to header lists

f7aec62

Second review:

8d8feab

- Distinguishes between precipitation rate and accumulation

Second review:

bf68e4c

- Raises a warning if the coordinate reference system is incomplete

Second review:

60ed07e

- Expands doc-strings for coord system functions

Second review:

f95a8f1

- Rearranges warning message to make it easier to read.

MoseleyS and others added 5 commits May 22, 2020 15:25

Second review:

f2a5dc6

- Moves origin_corner() to before horizontal_grid()

Adding unit tests for unit function

66079d8

Merge pull request #1 from fionaRust/nimrod_file_format

00dfa35

Adding unit tests for unit function

Removed unnecessary dictionaries from name function

0178eeb

Merge pull request #2 from fionaRust/nimrod_file_format

761d2d1

Removed unnecessary dictionaries from name function

fionaRust reviewed May 27, 2020

View reviewed changes

pp-mo mentioned this pull request May 27, 2020

Support loader-specific args in iris load functions. #3720

Closed

MoseleyS added 4 commits May 28, 2020 10:09

Improved comments for second review

454d57a

Merge branch 'nimrod_file_format' of github.com:MoseleyS/iris into ni…

ffca927

…mrod_file_format * 'nimrod_file_format' of github.com:MoseleyS/iris: Removed unnecessary dictionaries from name function Adding unit tests for unit function

Adds kwarg to allow user to skip known meta-data inconsistencies.

e6f70e0

Adds test for kwarg to allow user to skip known meta-data inconsisten…

a1519b3

…cies.

pp-mo reviewed May 28, 2020

View reviewed changes

MoseleyS added 2 commits May 28, 2020 14:58

Blacker code

3a117ce

fionaRust approved these changes May 29, 2020

View reviewed changes

pp-mo merged commit a07800c into SciTools:master Jun 4, 2020

tkknight pushed a commit to tkknight/iris that referenced this pull request Jun 29, 2020

Nimrod file format (SciTools#3647)

35a5c5f

dmentipl mentioned this pull request May 28, 2021

Add support for Iris 3.0 metoppv/improver#1507

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nimrod file format #3647

Nimrod file format #3647

MoseleyS commented Jan 27, 2020 •

edited

Loading

pp-mo commented May 22, 2020

fionaRust left a comment

fionaRust May 26, 2020

MoseleyS May 28, 2020

fionaRust May 29, 2020

fionaRust May 26, 2020

MoseleyS May 28, 2020

fionaRust May 26, 2020

MoseleyS May 28, 2020

fionaRust May 26, 2020

fionaRust May 26, 2020

fionaRust May 26, 2020

MoseleyS May 28, 2020

fionaRust May 26, 2020

MoseleyS May 28, 2020

pp-mo commented May 27, 2020

MoseleyS commented May 28, 2020

MoseleyS commented May 28, 2020

pp-mo May 28, 2020 •

edited

Loading

pp-mo May 28, 2020 •

edited

Loading

pp-mo commented May 28, 2020

fionaRust left a comment

pp-mo commented Jun 4, 2020

Nimrod file format #3647

Nimrod file format #3647

Conversation

MoseleyS commented Jan 27, 2020 • edited Loading

pp-mo commented May 22, 2020

fionaRust left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pp-mo commented May 27, 2020

MoseleyS commented May 28, 2020

MoseleyS commented May 28, 2020

pp-mo May 28, 2020 • edited Loading

Choose a reason for hiding this comment

pp-mo May 28, 2020 • edited Loading

Choose a reason for hiding this comment

pp-mo commented May 28, 2020

fionaRust left a comment

Choose a reason for hiding this comment

pp-mo commented Jun 4, 2020

MoseleyS commented Jan 27, 2020 •

edited

Loading

pp-mo May 28, 2020 •

edited

Loading

pp-mo May 28, 2020 •

edited

Loading