Skip to content

Commit

Permalink
add SYNOP support/pipeline (#304)
Browse files Browse the repository at this point in the history
  • Loading branch information
tomkralidis committed Jan 20, 2023
1 parent 6293452 commit 21e0361
Show file tree
Hide file tree
Showing 29 changed files with 809 additions and 25 deletions.
9 changes: 9 additions & 0 deletions .github/workflows/tests-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,15 @@ jobs:
python3 wis2box-ctl.py execute wis2box metadata discovery publish $DISCOVERY_METADATA
python3 wis2box-ctl.py execute wis2box data add-collection $DISCOVERY_METADATA
python3 wis2box-ctl.py execute wis2box data ingest -th $TOPIC_HIERARCHY -p $TEST_DATA
- name: add Romania data 🇷🇴
env:
TOPIC_HIERARCHY: rou.rnimh.data.core.weather.surface-based-observations.synop
DISCOVERY_METADATA: /data/wis2box/metadata/discovery/rou-synoptic-weather-observations.yml
TEST_DATA: /data/wis2box/observations/romania
run: |
python3 wis2box-ctl.py execute wis2box metadata discovery publish $DISCOVERY_METADATA
python3 wis2box-ctl.py execute wis2box data add-collection $DISCOVERY_METADATA
python3 wis2box-ctl.py execute wis2box data ingest -th $TOPIC_HIERARCHY -p $TEST_DATA
- name: sync stations 🔄
run: |
sleep 15
Expand Down
5 changes: 4 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,12 @@ RUN echo 'deb http://deb.debian.org/debian bookworm main' >> /etc/apt/sources.li
RUN apt-get update -y \
&& apt-get install -y -t bookworm libeccodes-data \
&& apt-get install -y ${DEBIAN_PACKAGES} \
# install wis2box dependencies
# install wis2box data pipeline dependencies
&& pip3 install --no-cache-dir https://github.com/wmo-im/csv2bufr/archive/master.zip \
&& pip3 install --no-cache-dir https://github.com/wmo-im/bufr2geojson/archive/refs/tags/v0.4.1.zip \
&& pip3 install --no-cache-dir https://github.com/wmo-im/pymetdecoder/archive/master.zip \
&& pip3 install --no-cache-dir https://github.com/wmo-im/synop2bufr/archive/main.zip \
# install geopython tooling
&& pip3 install --no-cache-dir https://github.com/geopython/pygeometa/archive/master.zip \
# install shapely
&& pip3 install cython pygeos==0.13 \
Expand Down
2 changes: 1 addition & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
project = 'WIS2 in a box'
author = 'World Meteorological Organization (WMO)'
license = 'This work is licensed under a Creative Commons Attribution 4.0 International License' # noqa
copyright = '2021-2022, ' + author + ' ' + license
copyright = '2021-2023, ' + author + ' ' + license

# The full version, including alpha/beta/rc tags

Expand Down
3 changes: 2 additions & 1 deletion docs/source/reference/how-wis2box-works.rst
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ wis2box is built on free and open source (FOSS) technology.
Storage,data and metadata storage,`MinIO`_ `Elasticsearch`_,`S3`_
Internal Message Broker,PubSub,`mosquitto`_, `MQTT`_
Public Message Broker,PubSub,`mosquitto`_, `MQTT`_
Data Management,data processing and publishing,`ecCodes`_ `csv2bufr`_ `bufr2geojson`_ `OWSLib`_ `pygeometa`_,:abbr:`WCMP (WMO Core Metadata Profile)` :abbr:`WMDR (WIGOS Metadata Record)`
Data Management,data processing and publishing,`ecCodes`_ `csv2bufr`_ `bufr2geojson`_ `synop2bufr`_ `OWSLib`_ `pygeometa`_,:abbr:`WCMP (WMO Core Metadata Profile)` :abbr:`WMDR (WIGOS Metadata Record)`
API Application,data discovery and access, `pygeoapi`_,`OGC API`_
Web Application,data discovery and visualization,`Vue.js`_ `Leaflet`_,`OGC API`_

Expand All @@ -91,6 +91,7 @@ wis2box is built on free and open source (FOSS) technology.
.. _`OGC API`: https://ogcapi.ogc.org
.. _`ecCodes`: https://confluence.ecmwf.int/display/ECC
.. _`csv2bufr`: https://csv2bufr.readthedocs.io
.. _`synop2bufr`: https://github.com/wmo-im/synop2bufr
.. _`bufr2geojson`: https://github.com/wmo-im/bufr2geojson
.. _`MinIO`: https://dosc.min.io
.. _`S3`: https://en.wikipedia.org/wiki/Object_storage
Expand Down
3 changes: 3 additions & 0 deletions docs/source/reference/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ Publish test discovery metadata:
wis2box metadata discovery publish $WIS2BOX_DATADIR/metadata/discovery/mwi-surface-weather-observations.yml
wis2box metadata discovery publish $WIS2BOX_DATADIR/metadata/discovery/ita-surface-weather-observations.yml
wis2box metadata discovery publish $WIS2BOX_DATADIR/metadata/discovery/dza-surface-weather-observations.yml
wis2box metadata discovery publish $WIS2BOX_DATADIR/metadata/discovery/rou-synoptic-weather-observations.yml
Setup observation collections from discovery metadata:
Expand All @@ -52,6 +53,7 @@ Setup observation collections from discovery metadata:
wis2box data add-collection $WIS2BOX_DATADIR/metadata/discovery/mwi-surface-weather-observations.yml
wis2box data add-collection $WIS2BOX_DATADIR/metadata/discovery/ita-surface-weather-observations.yml
wis2box data add-collection $WIS2BOX_DATADIR/metadata/discovery/dza-surface-weather-observations.yml
wis2box data add-collection $WIS2BOX_DATADIR/metadata/discovery/rou-synoptic-weather-observations.yml
Ingest data using the data ingest command to push data to the ``wis2box-incoming`` bucket:

Expand All @@ -60,6 +62,7 @@ Ingest data using the data ingest command to push data to the ``wis2box-incoming
wis2box data ingest --topic-hierarchy mwi.mwi_met_centre.data.core.weather.surface-based-observations.synop --path $WIS2BOX_DATADIR/observations/malawi
wis2box data ingest --topic-hierarchy ita.roma_met_centre.data.core.weather.surface-based-observations.synop --path $WIS2BOX_DATADIR/observations/italy
wis2box data ingest --topic-hierarchy dza.alger_met_centre.data.core.weather.surface-based-observations.synop --path $WIS2BOX_DATADIR/observations/algeria
wis2box data ingest --topic-hierarchy rou.rnimh.data.core.weather.surface-based-observations.synop --path $WIS2BOX_DATADIR/observations/romania
Publish stations:
Expand Down
2 changes: 1 addition & 1 deletion docs/source/reference/running/data-mappings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Each plugin is based on the file extension to be detected and processed, with th
- ``notify``: whether the plugin should publish a data notification
- ``template``: additional argument allowing a mapping template name to be passed to the plugin
- ``file-pattern``: additional argument allowing a file pattern to be passed to the plugin
- ``buckets``: the name(s) of the storage bucket(s) that data should be saved to (See :ref:`configuration` for more informaiton on buckets)
- ``buckets``: the name(s) of the storage bucket(s) that data should be saved to (See :ref:`configuration` for more information on buckets)

The default data mapping can be overriden by user-defined data mappings with the following steps:

Expand Down
10 changes: 8 additions & 2 deletions docs/source/reference/running/data-pipeline-plugins.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ Default pipeline plugins

wis2box provides a number of data pipeline plugins which users can be used "out of the box".


``wis2box.data.csv2bufr.ObservationDataCSV2BUFR``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand All @@ -33,12 +32,19 @@ This plugin converts BUFR observation data into GeoJSON using ``bufr2geojson``.
can be used to filter on incoming data based on a regular expression. Consult the bufr2geojson documentation
for more information on configuration and templating.


``wis2box.data.geojson.ObservationDataGeoJSON``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This plugin is for the purposes of publishing GeoJSON data to the API.

``wis2box.data.synop2bufr.SYNOP2BUFR``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This plugin converts SYNOP ASCII data into BUFR using ``synop2bufr``. A ``file-pattern`` can be used
to filter on incoming data based on a regular expression. Note that the regex must contain two groups
(for year and month), which are used as part of synop2bufr processing. Consult the synop2bufr documentation
for more information.

.. _`csv2bufr`: https://csv2bufr.readthedocs.io
.. _`bufr2geojson`: https://github.com/wmo-im/bufr2geojson
.. _`synop2bufr`: https://github.com/wmo-im/synop2bufr
4 changes: 2 additions & 2 deletions docs/source/reference/running/index.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _running:

Workflows
=========
Running
=======

wis2box workflows can be categorized as design time (interactive) or runtime (automated).

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
wis2box:
retention: P30D
topic_hierarchy: rou.rnimh.data.core.weather.surface-based-observations.synop
country: rou
centre_id: rnimh

mcf:
version: 1.0

metadata:
identifier: urn:x-wmo:md:rou:rnimh:synoptic-weather-observations
hierarchylevel: dataset

identification:
title: Synoptic weather observations from Romania
abstract: Synoptic weather observations from Romania
dates:
creation: 2023-01-18
keywords:
default:
keywords:
- synoptic weather
- temperature
- observations
wmo:
keywords:
- weatherObservations
keywords_type: theme
vocabulary:
name: WMO Category Code
url: https://github.com/wmo-im/wcmp-codelists/blob/main/codelists/WMO_CategoryCode.csv
extents:
spatial:
- bbox: [20.2201924985,43.6884447292,29.62654341,48.2208812526]
crs: 4326
temporal:
- begin: 2023-01-17
end: null
resolution: P6H
url: https://example.org/malawi-surface-weather-observations
wmo_data_policy: core

contact:
pointOfContact: &contact_poc
organization: Romanian National Meteorological Administration
url: https://www.meteoromania.ro
individualname: Firstname Lastname
positionname: Position Name
phone: +40 21 318 32 40
fax: +40 21 316 31 43
address: Building A, Șoseaua București-Ploiești 9
city: Bucharest
administrativearea: Municipality of Bucharest
postalcode: 013686
country: Romania
email: [email protected]
hoursofservice: 0700h - 1500h UTC
contactinstructions: email

distributor: *contact_poc
23 changes: 23 additions & 0 deletions tests/data/metadata/station/station_list.csv
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,26 @@ EL-BAYADH,0-20000-0-60550,60550,Land (fixed),33.71966,1.09484,1361,Algeria,1
USTICA,0-20000-0-16400,16400,Land (fixed),38.7072222222,13.1772222222,242,Italy,6
GROSSETO,0-20000-0-16206,16206,Land (fixed),42.7480555556,11.0588888889,5.4,Italy,6
BENI OUNIF,0-12-0-08BECCN60577,60577,Land (fixed),32.05138,-1.26527,830,Algeria,1
OCNA SUGATAG,0-20000-0-15015,15015,Land (fixed),47.7770616258,23.9404602638,503,Romania,6
BOTOSANI,0-20000-0-15020,15020,Land (fixed),47.7356532437,26.6455501701,161,Romania,6
IASI,0-20000-0-15090,15090,Land (fixed),47.163333333,27.6272222222,74.29,Romania,6
CEAHLAU TOACA,0-20000-0-15108,15108,Land (fixed),46.9775099973,25.9499399749,1897,Romania,6
CLUJ-NAPOCA,0-20000-0-15120,15120,Land (fixed),46.7777705044,23.5713052939,410,Romania,6
BACAU,0-20000-0-15150,15150,Land (fixed),46.5577777778,26.8966666667,174,Romania,6
MIERCUREA CIUC,0-20000-0-15170,15170,Land (fixed),46.3713166568,25.7726166755,661,Romania,6
ARAD,0-20000-0-15200,15200,Land (fixed),46.1335163958,21.3536215174,116.59,Romania,6
DEVA,0-20000-0-15230,15230,Land (fixed),45.8649230138,22.898806236,240,Romania,6
SIBIU,0-20000-0-15260,15260,Land (fixed),45.79018,24.036245,450,Romania,6
VARFU OMU,0-20000-0-15280,15280,Land (fixed),45.4457927989,25.456690976,2504,Romania,6
CARANSEBES,0-20000-0-15292,15292,Land (fixed),45.41667,22.22917,241,Romania,6
GALATI,0-20000-0-15310,15310,Land (fixed),45.4729181384,28.0323010582,69,Romania,6
TULCEA,0-20000-0-15335,15335,Land (fixed),45.1905064849,28.8241607619,4.36,Romania,6
RAMNICU VALCEA,0-20000-0-15346,15346,Land (fixed),45.0888211225,24.3628139123,237,Romania,6
BUZAU,0-20000-0-15350,15350,Land (fixed),45.1326632857,26.8517319231,97,Romania,6
SULINA,0-20000-0-15360,15360,Land (fixed),45.1623111,29.7268286,12.69,Romania,6
DROBETA-TURNU SEVERIN,0-20000-0-15410,15410,Land (fixed),44.6264587019,22.6260737132,77,Romania,6
BUCURESTI BANEASA,0-20000-0-15420,15420,Land (fixed),44.5104330044,26.0781904077,90,Romania,6
CRAIOVA,0-20000-0-15450,15450,Land (fixed),44.3101404313,23.8669847441,192,Romania,6
CALARASI,0-20000-0-15460,15460,Land (fixed),44.2057385289,27.3383080718,18.72,Romania,6
ROSIORII DE VEDE,0-20000-0-15470,15470,Land (fixed),44.1072133362,24.9787400713,102.15,Romania,6
CONSTANTA,0-20000-0-15480,15480,Land (fixed),44.2138143888,28.6455646839,12.8,Romania,6
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
SMRO01 YRBK 171200 CCA
AAXX 17121
15108 01/92 92514 11028 21028 37901 48315 50001 69941 77174 333 48014
55300 0//// 20270 3//// 69947 91020 911// 92727 92913 96047=

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
SMRO01 YRBK 171200 CCB
AAXX 17121
15280 01/90 92518 11047 21054 37334 47872 51004 60001 74143 333 49070
55300 0//// 20000 3//// 60007 91026 911// 92956=

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
SMRO01 YRBK 171200
AAXX 17121
15015 01597 71702 10057 20036 39390 42628 50004 60021 78082 87300 333
4/000
55304 0//// 20643 3//// 69977 91003 91108=
15020 02597 61303 10104 20040 39783 49976 58007 60001 83570 333 4/000
55308
0//// 21085 3//// 60007 91005 91106=
15090 02597 72003 10109 20036 39904 49994 56010 60001 83571 333 4/000
55301
00256 20589 30542 60007 91005 91107=
15108 01/92 92514 11028 21028 37901 48315 50001 69941 77174 333 48013
55300
0//// 20270 3//// 69947 91020 911// 92727 92913 96047=
15120 02598 61904 10106 20016 39496 49978 50003 60001 83360 333 55309
00649
21413 30243 60007 91007 91107=
15150 02597 61604 10091 20036 39782 49991 57013 60001 86800 333 4/000
55308
0//// 20986 3//// 60007 91006 91109=
15170 01597 81701 10050 20031 39238 42652 53007 69961 76162 885// 333
55300
0//// 20206 3//// 69937 91003 91104=
15200 02598 51707 10111 20062 39840 49980 58006 60001 81140 333 4/000
55310
0//// 21428 3//// 60007 91010 91111=
15230 01598 70101 10089 20054 39714 40002 58007 60011 70262 87500 333
55303
00347 20712 30640 60007 91002 91102=
15260 05599 21502 10088 20029 39467 49998 50004 60001 7000/ 82800 333
55309
0//// 21214 3//// 60007 91003 91104 95000=
15280 01/90 92518 11047 21054 37334 47872 51004 60001 74143 333 49070
55300
0//// 20000 3//// 60007 91026 911// 92956=
15292 01597 61505 10074 20048 39713 40004 58010 60021 70182 86300 333
55302
0//// 20637 3//// 60007 91008 91108 95090=
15310 02998 71903 10111 20066 39942 40026 55008 60001 82038 333 4/000
55304
00448 20862 30571 60007 91007 91110=
15335 02597 51705 10124 20085 30028 40035 57013 60001 83530 333 55309
0////
21651 3//// 60007 91008 91110=
15346 01596 73401 10059 20047 39746 40035 58003 60021 76162 85570 333
4/000
55300 0//// 20415 3//// 69947 91002 91103=
15350 02997 32204 10103 20057 39906 40025 57020 60001 80002 333 4/000
55308
0//// 20913 3//// 60007 91006 91106=
15360 02997 61910 10098 20077 30030 40046 57009 60001 86070 222//
06070 20503
333 55303 ///// 20758 3//// 60007 91011 91112 92447=
15410 01597 80302 10066 20045 39934 40030 57013 60031 70262 8457/ 333
4/000
55301 0//// 20600 3//// 60017 91004 91106=
15420 01597 62101 10127 20087 39912 40021 58011 69911 70262 83570 333
55305
0//// 20850 3//// 60007 91003 91103=
15450 01596 70203 10078 20054 39792 40025 50001 69911 71086 84530 333
55306
00537 21202 30592 69907 91004 91105=
15460 01997 72106 10175 20069 30002 40026 57014 69911 70282 87070 333
4/000
55310 0//// 21460 3//// 60007 91011 91111=
15470 02997 51902 10141 20082 39897 40019 58008 60001 85050 333 55310
0////
21469 3//// 60007 91003 91103=
15480 05997 42104 10146 20092 30023 40040 57015 60001 7000/ 82031
222// 06078
2//// 333 55309 00621 21275 30948 60007 91010 91112 92427=

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
SMRO01 YRBK 171200
AAXX 17121
15015 01597 71702 10057 20036 39390 42628 50004 60021 78082 87300 333
4/000
55304 0//// 20643 3//// 69977 91003 91108=
15020 02597 61303 10104 20040 39783 49976 58007 60001 83570 333 4/000
55308
0//// 21085 3//// 60007 91005 91106=
15090 02597 72003 10109 20036 39904 49994 56010 60001 83571 333 4/000
55301
00256 20589 30542 60007 91005 91107=
15108 01/92 92514 11028 21028 37901 48315 50001 69941 77174 333 48013
55300
0//// 20270 3//// 69947 91020 911// 92727 92913 96047=
15120 02598 61904 10106 20016 39496 49978 50003 60001 83360 333 55309
00649
21413 30243 60007 91007 91107=
15150 02597 61604 10091 20036 39782 49991 57013 60001 86800 333 4/000
55308
0//// 20986 3//// 60007 91006 91109=
15170 01597 81701 10050 20031 39238 42652 53007 69961 76162 885// 333
55300
0//// 20206 3//// 69937 91003 91104=
15200 02598 51707 10111 20062 39840 49980 58006 60001 81140 333 4/000
55310
0//// 21428 3//// 60007 91010 91111=
15230 01598 70101 10089 20054 39714 40002 58007 60011 70262 87500 333
55303
00347 20712 30640 60007 91002 91102=
15260 05599 21502 10088 20029 39467 49998 50004 60001 7000/ 82800 333
55309
0//// 21214 3//// 60007 91003 91104 95000=
15280 01/90 92518 11047 21054 37334 47872 51004 60001 74143 333 49070
55300
0//// 20000 3//// 60007 91026 911// 92956=
15292 01597 61505 10074 20048 39713 40004 58010 60021 70182 86300 333
55302
0//// 20637 3//// 60007 91008 91108 95090=
15310 02998 71903 10111 20066 39942 40026 55008 60001 82038 333 4/000
55304
00448 20862 30571 60007 91007 91110=
15335 02597 51705 10124 20085 30028 40035 57013 60001 83530 333 55309
0////
21651 3//// 60007 91008 91110=
15346 01596 73401 10059 20047 39746 40035 58003 60021 76162 85570 333
4/000
55300 0//// 20415 3//// 69947 91002 91103=
15350 02997 32204 10103 20057 39906 40025 57020 60001 80002 333 4/000
55308
0//// 20913 3//// 60007 91006 91106=
15360 02997 61910 10098 20077 30030 40046 57009 60001 86070 222//
06070 20503
333 55303 20758 3//// 60007 91011 91112 92447=
15410 01597 80302 10066 20045 39934 40030 57013 60031 70262 8457/ 333
4/000
55301 0//// 20600 3//// 60017 91004 91106=
15420 01597 62101 10127 20087 39912 40021 58011 69911 70262 83570 333
55305
0//// 20850 3//// 60007 91003 91103=
15450 01596 70203 10078 20054 39792 40025 50001 69911 71086 84530 333
55306
00537 21202 30592 69907 91004 91105=
15460 01997 72106 10175 20069 30002 40026 57014 69911 70282 87070 333
4/000
55310 0//// 21460 3//// 60007 91011 91111=
15470 02997 51902 10141 20082 39897 40019 58008 60001 85050 333 55310
0////
21469 3//// 60007 91003 91103=
15480 05997 42104 10146 20092 30023 40040 57015 60001 7000/ 82031
222// 06078
2//// 333 55309 00621 21275 30948 60007 91010 91112 92427=

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
SMRO01 YRBK 171800 CCA
AAXX 17181
15090 02997 12101 10084 20048 39928 40020 51019 60002 81030 333 10122
20059 31010 4/000 55300 10143 20000 30000 60007 91004 91104=

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
SMRO01 YRBK 171800 CCB
AAXX 17181
15280 01/90 92012 11041 21047 37349 47889 50004 60002 74143 333 11041
21059
3//// 49075 55300 0//// 20000 3//// 60007 91020 911// 92818 92946=

Expand Down
Loading

0 comments on commit 21e0361

Please sign in to comment.