censor_thresh in plot_data_plane and grid_stat

[fearon10]

Hi,

My question is almost identical to discussion item#1389 [https://github.com//discussions/1389].

I have a grid I'm reading into MET with python embedding. It reads into grid_stat and plot_data_plane without issue. However, I would like to use the censor_thresh option in grid_stat. When I set the option, I do not receive an error, but the stat results indicate that my censor_thresh setting was ignored by grid_stat.
Therefore, based on 1389, I am trying to first visualize the impact of my censor_thresh setting with plot_data_plane. However, I must have a syntax error as the censor_thresh criteria doesn't appear to work as shown in 1389.

This works without issue:
singularity exec
met_v12.0.0-beta1.simg plot_data_plane PYTHON_NUMPY nep.ps 'name="read_neptune2.py -f neptune_quarterdeg_E0096P6L101_20210301T00_PT000000H00M00S.hdf5 -m NEPTUNE -l 100000 -lt pressure -v air_temperature -of 2021030100_sfc.nc -ov sp -om ERA5";'

This does not work, only difference is that the censor_thresh option is added
singularity exec
met_v12.0.0-beta1.simg plot_data_plane PYTHON_NUMPY nep.ps 'name="read_neptune2.py -f neptune_quarterdeg_E0096P6L101_20210301T00_PT000000H00M00S.hdf5 -m NEPTUNE -l 100000 -lt pressure -v air_temperature -of 2021030100_sfc.nc -ov sp -om ERA5";' censor_thresh = eq-9999; -v 10;

*** Model Evaluation Tools (METV12.0.0) ***

Usage: plot_data_plane
input_filename
output_filename
field_string
[-color_table color_table_name]
[-plot_range min max]
[-title title_string]
[-log file]
[-v level]

Thanks,
Matt

[CPKalb]

Hi Matt:

I see two things in the second command above that may be causing the issue. The first is that although censor_thresh is added in the second command, censor_val is not part of that command. For censor_thresh to work, you need both the censor_thresh and censor_val set. The censor_thresh provides a threshold for data that you want to censor, while censor_val is the value that these values should be changed to. Since no censor_val was provided, the censor_thresh option should produce no change.

The second is that the censor_thresh and censor_val need to be inside the single quote with the name variable for the Plot-Data-Plane command.

If you want to set your missing data to 0, as is done in discussion 1389, then I think you need the following command to do that:

singularity exec
met_v12.0.0-beta1.simg plot_data_plane PYTHON_NUMPY nep.ps 'name="read_neptune2.py -f neptune_quarterdeg_E0096P6L101_20210301T00_PT000000H00M00S.hdf5 -m NEPTUNE -l 100000 -lt pressure -v air_temperature -of 2021030100_sfc.nc -ov sp -om ERA5"; censor_thresh = eq-9999; censor_val = 0;' -v 5

If you run the command as above, does this produce the desired result on your end?

Thanks,
Christina

[CPKalb]

Also, would you be willing to let us know where you are from, so we can add the requestor label to this discussion?

Thanks,
Christina

[fearon10]

Thank you, Christina. Using your two suggestions and the command provided, I was able to get plot_data_plane to work. Grid_stat also worked, and I can see that censor points are being excluded in the stat calculations.

Ultimately, I would like to use the censor_thresh and val method with grid_stat to restrict points entering the stats calculation. Using this approach instead of a binary mask (applied based on the same criteria) yields slightly different results in the number of points retained in the stats calculation. I'm not exactly sure why a difference exists. I must be missing some little nuance.

Thanks again.
p.s., my affiliation is the SAIC, Monterey CA.

Matt

[CPKalb]

Hi Matt:

Great to hear it worked and thanks for providing you affiliation!

In regards to the minor differences, I can think of two things that I have seen cause small differences previously. There may be others, but these would be a place to start to check if they result in any changes to the output.

I'm assuming here that your mask file is a 0/1 binary and that it's on the same grid as what you are using to verify. If it's not on the same grid, this can cause minor changes so would be something to look at. Otherwise, checking the two things below would be a good starting point:

1. Are you doing any regridding in your Grid-Stat run? If you are not doing regridding, then this shouldn't be an issue. But if you are, then it's important to note that censoring is applied prior to regridding. Any time you regrid, this can cause small differences in the values that have been reset, especially if you are regridding with a method that has a width greater than 1. There is a way to apply the censor_thresh and censor_val after regridding has taken place. In MET this would be done by setting censor_thresh and censor_val in the regrid dictionary. In METplus it can be accomplished by using GRID_STAT_REGRID_CENSOR_THREHS and GRID_STAT_REGRID_CENSOR_VAL. I can provide more information if this applies to your data.

2. Another thing I have seen is slight differences in data packing. By this, I mean that I've seen MET store data values slightly differently from what they appear in a dataset. So, for example a value of 10 might be stored as 9.99999999999, or 10.0000000000001. Obviously with rounding and precision we usually don't see differences when this happens, but when thresholding values, this can cause differences. One quick way to test if this may be an issue would be to change your censor thresh above to something like censor_thresh = lt-9998.9999; or something similar. If this changes your output, we could have a software engineer verify that the issue is data packing.

Christina

[fearon10]

Hi Christina,
Thanks for all the great feedback regarding the reasons for possible differences, using a mask vs. censor_thresh.

I am using a binary mask 0/1 and there is no regridding (same grid) being performed. I also did test using a censor_thresh as you suggested "censor_thresh = lt-9000", but the same differences remained.

I'll keep tinkering with it. I'm probably missing a small item.

Thanks again for all the help.
Matt

[CPKalb]

How large of a difference are you seeing in the total value?

Christina

[fearon10]

Hi Christina,
After some additional digging, I found the source of my error. I am able to now get the same stats answer with censor_thresh vs. a mask. The number of points retained also matches between methods.
Thanks again,
Matt

[CPKalb]

Hi Matt:

Great! I'm going to go ahead and mark this as answered and close the discussion. If you have additional issues, feel free to start up a new discussion.

Christina