Incompatibility for CTC line type since MET v11.0.0

[daniel-lagrava]

Hello,

I wrongly created an issue on the METviewer project, could you please delete it? (https://github.com/dtcenter/METviewer/issues/447)

I am bringing that discussion here:
CTC line type had 15 columns for files generated with MET v10.X:

VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FY_OY FY_ON FN_OY FN_ON

Since MET v11.0.0, there is an additional column (EC_VALUE) as it can be seen here: https://met.readthedocs.io/en/latest/Users_Guide/point-stat.html?highlight=EC_VALUE#id7

When trying to load those sort of files using mv_load_mv_load.sh the parser crashes:

edu.ucar.metviewer.DatabaseException: caught SQLException calling executeBatch: (conn=25) Could not set parameter at position 16 (values was '0.5')
Query - conn:25(M)  - "INSERT INTO line_data_ctc VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"

An example of a faulty line is:

VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD OBS_VALID_BEG   OBS_VALID_END   FCST_VAR FCST_UNITS FCST_LEV OBS_VAR OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FY_OY FY_ON FN_OY FN_ON EC_VALUE
V11.0.0 TEST  NA   060000    20221008_060000 20221010_060000 000000   20221008_055500 20221010_055500 QOP6Hr   kg/m^2     A6       APCP_06 kg/m^2    A06     ADPSFC K0E0    NEAREST     1           >0          >0         NA         NA    CTC           3     1     0     2     0      0.5

We are using an older version of METviewer (v4.1) but looking at the source code, the latest version and the latest release have the limitation to 15 columns.

Thanks a lot for your help!

[TatianaBurek]

Output from MET v11.0.0 can be loaded only by the compatible version of MET analyses tools:
https://dtcenter.org/community-code/metplus/coordinated-metplus-version-5-0
It means that you need to install METviewer v5.0 , METplotpy, METcalcpy and METdataio v2.0.0 and update METviewer scripts with correct paths to Python
To create a new database schema please use this script:
https://github.com/dtcenter/METdataio/blob/main_v2.0/METdbLoad/sql/mv_mysql.sql

To load the data please use this script:
https://github.com/dtcenter/METviewer/blob/main_v5.0/bin/mv_load.sh
It calls Python script from METdataio

[daniel-lagrava]

Thanks for your reply. I didn't see that coordinated release. I will give it a try! We have a number of other databases that we created for MET v10, are you aware of any backward incompatibilities when doing the upgrade to v5.0? Cheers!

[TatianaBurek]

Database schema that was provided in METplus v4.1 was changed in METplus v5.0 because of the addition of new statistics in MET v11.0.
If you created databases with METviewer 4.1 and METdatadb v1.1 and would like to use them with METplus v5.0 you need to run an update script on those databases. This script updates the tables and adds new columns. Let me know if this is the case and I will provide this script.

[daniel-lagrava]

Hello, thanks again for your reply. We have databases that we created with the earlier releases, so it would be greatly appreciated to have the script to update the tables as needed. Cheers!

[daniel-lagrava]

Hello @TatianaBurek , just a quick message to bump this, if you could please provide the script to update the METplus v4.1 databases to be used with the new version. Thanks a lot!

[TatianaBurek]

@daniel-lagrava-spire
Here is the script. You need to remove .txt extension that was added to comply with GitHub Discussions format
update_for_5_0.sql.txt

[daniel-lagrava]

@TatianaBurek thanks a lot for that! This closes all my questions about this subject, I appreciate you taking the time to answer them.