Skill Score Diagram for Rapid Intensification

[YanJin-NOAA]

Hi!

I wonder if MET-TC tool can plot the skill score of the rapid intensification like this figure?

I found that MET-Stat tool can compare the occurrence of rapid intensification between adeck and bdeck, and output a table. So I wonder if I can use MET to plot the figure? Or I need to write a script to plot the figure by myself?

Thanks very much.
Best,
Yan

[JohnHalleyGotway]

@YanJin-NOAA I see you have a question about plotting categorical statistics of Tropical Cyclone rapid intensification events. Thanks for including an image to illustrate your question. This type of plot is called a "Performance Diagram" and it's used to summarize multiple statistics from a 2x2 contingency table on the same plot.

Unfortunately, I believe the answer to your question is no. At this time, there is not an easy way to create the plot you're after. But let me tell you about 2 related details:

(1) The METviewer software IS able to create performance diagrams. Looking in the METviewer GUI, the tab named "Perf" is used to create performance diagrams. So if the RIRW contingency table counts lived in the database for METviewer, you could create that plot.

(2) The TC-Stat "-job RIRW" is presumably the job you're running. That's the job that inspects the TC-Pairs output looking for the occurrence of rapid intensification and defining contingency table counts and statistics. However, the contingency table output it writes is a custom format and looks something like this:

COL_NAME: ATIME  BTIME  AEXACT BEXACT ATHRESH BTHRESH WINDOW_BEG WINDOW_END TOTAL FY_OY FY_ON FN_OY FN_ON
RIRW_CTC: 240000 240000 true   true   <=-15   <=-15   -120000    120000        42     2     0     2    38

While all the information is present, it does not follow the full .stat file format. So it can't be easily loaded into METviewer. We didn't initially format it like that because it wasn't obvious how columns like ATIME, BTIME, AEXACT, BEXACT, ATHRESH, BTHRESH, WINDOW_BEG, and WINDOW_END should be mapped into the existing .stat header columns.

Perhaps we should revisit this decision and update TC-Stat to actually write CTC and CTS line types that can be loaded into a database for METviewer?

Hope that helps clarify.

[YanJin-NOAA]

Hi John,

Thanks very much for your detailed instruction. I will run the script by myself and learn to read the output file.

Best
Yan

[YanJin-NOAA]

Hi John,

I am comparing the information in the performance diagram and the TC-Stat -rirw output file.

The x-axis of the performance diagram is "Success Ratio", which can be calculated by 1-FAR, where FAR is False Alarm Rate, and it is included in the rirw output file.

The y-axis of the performance diagram is "probability of detection, POD", which is defined by hit events / (hit events + miss events). In the rirw output file there are several columns containing "POD", such as "PODY", "PODY_NCL", "PODY_NCU","PODN", etc. I wonder which one should I use to plot the performance diagram? (I assume I should use "PODY" related column since PODN is almost 1)

Thanks very much for your time and help.
Best,
Yan

[JohnHalleyGotway]

Yan, Appendix C of the MET User's Guide lists the equations for each of the statistics.

And I'm looking at this Performance Diagram reference:
https://www.cawcr.gov.au/projects/verification/Roebber/PerformanceDiagram.html

For the Y-axis, yes "PODY" is how MET writes the "Probability of Detection" (we put the Y at the end to differentiate between PODN).

The X-axis is a little tricky. There are 2 very similar stats...

• False Alarm Rate (i.e. also called Probability of False Detection, or POFD)
• False Alarm Ratio (called FAR in the MET output)

I see that the METviewer plotting script for performance diagrams, plots PODY versus 1-FAR from MET. So that would be probability of detection versus success ratio (1 minus false alarm ratio (not rate)).

I find the whole "ratio" vs "rate" confusion very unfortunate!

[TaraJensen]

@JohnHalleyGotway @bikegeek @fisherhucar @TatianaBurek - Is there anyway we can show Yan how to run
METplotpy/metplotpy/plots/performance_diagram/performance_diagram.py on ~/rirw_ctc_by_model.txt, even if it requires a few more lines to read the the output? We already have the code to plot the Performance Diagram - it seems kinda crazy to reinvent the wheel.

[TaraJensen]

@YanJin-NOAA - If you can wait until next Tue/Wed, we will hopefully have a solution for you that doesn't require you to write your own Performance Diagram plotting script.

[JohnHalleyGotway]

@TaraJensen, as we discussed in the METplus meeting today, I wrote up this development issue to enhance TC-Stat to write full .stat output files that could be loaded into a database for use by METviewer. I tentatively assigned it to MET version 10.1.0 but did not assign it to a specific development cycle or particular developer or scientist. Please feel free to take a look and update/modify it however you see fit.

Note that the TC-Stat -job rirw and STAT-Analysis -job ramp both have this similar situation. They both write categorical output to custom, job-specific formats that cannot be easily loaded into a database for use by METviewer.

The development issue covers TC-Stat, not STAT-Analysis. But we could write up a similar one for STAT-Analysis at some point if needed. But the difficulty is in the details... deciding how to shoehorn job-specific info into the existing STAT header columns in an intuitive way.

[YanJin-NOAA]

Hi Tara, I found a Python script to plot the performance diagram: https://gist.github.com/djgagne/64516e3ea268ec31fb34 I plan to modify the code to 1) read the TC-Stat -job rirw output file, and then 2) do the plotting. Currently the input data (model names, PODY, FAR) is hard coded and saved to a dictionary. The next step is to read the file, find the corresponding columns, and convert them to a dictionary or array. The plot looks like this: Thanks very much for your time and help. I feel it would be very helpful for the verification team if we can visualize the result using MET/METplus. Since TC-Stat has already generated the output file so we do need to do the statistics by ourselves. Best, Yan [image: image.png]

…

On Fri, Nov 12, 2021 at 6:02 PM TaraJensen ***@***.***> wrote: @YanJin-NOAA <https://github.com/YanJin-NOAA> - If you can wait until next Tue/Wed, we will hopefully have a solution for you that doesn't require you to write your own Performance Diagram plot. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1244 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATBLJOCFQCTUKONECEQFHL3ULWMI3ANCNFSM5HTE5YHQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[YanJin-NOAA]

I just realized that the plot was not attached in the last post. (I replied the email and attached the plot in the email)