NDBC buoy spectral files #2209
-
|
Hello, ASCII2NC can read standard NDBC buoy files, for example 41008.txt Would it be possible to expand it to read the spectral buoy files too? i.e. 41008.spec Many thanks, |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 2 replies
-
|
@DeannaSpindler-NOAA thanks for pointing us to a sample data file. I did try running this through ascii2nc and got an error message, as you might expect: So it won't read it directly out of the box. Looking at this line of the I'm wondering about the spectral buoy data (41008.spec). Does it follow the same paradigm? Is it always 15 columns provided in this pre-defined order or can there be variability depending on the source? Or does the format change over time? For now, using python embedding should be pretty simple for this data. But adding direct support for it should be pretty straight-forward too. I can think of 2 implementation options:
Which sounds like the better approach to you? @davidalbo, just tagging you on this discussion to loop you in. |
Beta Was this translation helpful? Give feedback.
-
|
Hello John,
I believe that the ASCII file columns are set for the specific file types,
so the standard met files will always have the same 19 columns, the
spectral files will always have the same 15 columns, etc.
I looked on the NDBC website (https://www.ndbc.noaa.gov/faq/measdes.shtml)
and there about 18 different realtime ASCII file types. Of those, only the
raw spectral wave data files (
https://www.ndbc.noaa.gov/data/realtime2/41008.data_spec for example), have
very different formats and won't conform as easily to the "number of
columns and column names" reader.
For the wave reporting buoys (
https://www.ndbc.noaa.gov/station_realtime.php?station=41008 for example),
there can be 10 types of ASCII files available, five of which conform to
the "number of columns and column names" format, and 5 that are raw
spectral wave data files. In the short-term I don't expect to use anything
other than the standard met and the spectral files, but other users may be
interested in the raw spectral files.
My feeling is that in the short-term it would be easier to define a -format
ndbc_spec to read the spectral files. In the long term it would be nice
for MET to be able to read any of the NDBC ASCII files, so having a reader
that can be used for any of the 17 different "number of columns and column
names" formats would be useful. The good news is that the two header lines
are there for 15 of the 17 different types of columnar data files - two of
them have only one header line. At some point you might consider adding a
separate ndbc_raw_spectra_handler for the raw spectral wave data, or find a
way to incorporate reading the raw spectral files into the handler you
already have. The raw spectral data files also have only one header line.
Best,
Deanna
…--
Deanna Spindler, PhD
Physical Scientist III
Lynker at NOAA/NWS/NCEP/Ocean Prediction Center
NOAA Center for Weather and Climate Prediction
On Mon, Jun 12, 2023 at 11:51 PM John Halley Gotway < ***@***.***> wrote:
@DeannaSpindler-NOAA <https://github.com/DeannaSpindler-NOAA> thanks for
pointing us to a sample data file. I did try running this through ascii2nc
and got an error message, as you might expect:
ascii2nc 41008.spec 41008.nc -format ndbc_standard
DEBUG 1: Start ascii2nc by johnhg(6088) at 2023-06-12 23:35:57Z cmd: bin/ascii2nc 41008.spec 41008.nc -format ndbc_standard
DEBUG 1: Config File: /Volumes/d1/projects/MET/MET_development/MET-develop/share/met/config/Ascii2NcConfig_default
WARNING:
WARNING: NdbcHandler::_readHeaderInfo() -> Unexpected number of header columns (15 != 19): 41008.spec
WARNING:
ERROR :
ERROR : ascii2nc-> encountered an error while reading input files!
ERROR :
So it won't read it directly out of the box.
Looking at this line
<https://github.com/dtcenter/MET/blob/cf9bdc29ddc2f64ba6221c83ec1db848ad446352/src/tools/other/ascii2nc/ndbc_handler.cc#L44>
of the ndbc_handler.cc file in MET, I see that the file format for the
NDBC buoy data is rather hard-coded. It expects 19 input columns in each
line, and expects the observations to be defined in exactly that
pre-defined order.
I'm wondering about the spectral buoy data (41008.spec). Does it follow
the same paradigm? Is it always 15 columns provided in this pre-defined
order or can there be variability depending on the source? Or does the
format change over time?
For now, using python embedding should be pretty simple for this data. But
adding direct support for it should be pretty straight-forward too.
I can think of 2 implementation options:
1. Add support for another hard-coded format -format ndbc_spec that
requires exactly 15 input columns (just like how -format ndbc_standard
requires 19).
2. Alternatively, we could generalize the -format ndbc_standard to
define the observation variable names and units based on the metadata
provided. Some column names would still be required (YY MM DD hh mm, for
example), but the number of data columns and corresponding unit strings
could vary. So we'd generalize the existing -format ndbc_standard to
handle both 41008.txt
<https://www.ndbc.noaa.gov/data/realtime2/41008.txt> and 41008.spec
<https://www.ndbc.noaa.gov/data/realtime2/41008.spec> input files. The
advantage would be making the code more flexible. The disadvantage would be
requiring those 2 header lines be present each time we read NDBC buoy data.
Which sounds like the better approach to you?
@davidalbo <https://github.com/davidalbo>, just tagging you on this
discussion to loop you in.
—
Reply to this email directly, view it on GitHub
<#2209 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AK6TDPXEYPWNDOOQHNBAMU3XK6TRPANCNFSM6AAAAAAZDVTNMA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Deanna, thanks for providing this information. I do think its worthwhile to pursue a more general solution rather than going adding several new hard-coded Do you need a short term python embedding workaround to read this spectral wave data? |
Beta Was this translation helpful? Give feedback.
-
|
If it's something that can be done quickly and easily, then sure! If not, I
can either wait or try to write one myself.
Thanks,
Deanna
…--
Deanna Spindler, PhD
Physical Scientist III
Lynker at NOAA/NWS/NCEP/Ocean Prediction Center
NOAA Center for Weather and Climate Prediction
On Tue, Jun 13, 2023 at 6:02 PM John Halley Gotway ***@***.***> wrote:
Deanna, thanks for providing this information. I do think its worthwhile
to pursue a more general solution rather than going adding several new
hard-coded -format types. I wrote this up in a new dtcenter/MET#2571
<dtcenter/MET#2571> issue. Please feel free to
comment on it as you see fit.
Do you need a short term python embedding workaround to read this spectral
wave data?
—
Reply to this email directly, view it on GitHub
<#2209 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AK6TDPSNTYILWMGWZFLSVB3XLCTMZANCNFSM6AAAAAAZDVTNMA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
I wanted to chime in here and see if you were able to start working on a Python embedding solution for your spectral wave ingest needs. There are many resources of existing METplus use cases utilizing Python scripts to ingest non-native MET datasets I can point you to if you're in need. If you're feeling too overwhelmed to start or took a swing at it and got stuck, I can take a look at the example spectral file and see if I can get something together for you sometime this week or next. |
Beta Was this translation helpful? Give feedback.
Deanna, thanks for providing this information. I do think its worthwhile to pursue a more general solution rather than going adding several new hard-coded
-formattypes. I wrote this up in a new dtcenter/MET#2571 issue. Please feel free to comment on it as you see fit.Do you need a short term python embedding workaround to read this spectral wave data?