Trollflow2 Satellite Image Generation
In this page you will find a quick but in depth guide on how to configure .YAML files (which are the base of the Trollflow2 system) so to obtain satellite imagery using Pytroll - Satpy.
For information on how to install Trollflow2 using Dockers, please refer to → Trollflow2 Installation Tutorial
First, we need to understand how to navigate inside the docker. A usual directory path to the configuration files is the following:
/home/user/dockers/pytroll/mnt/trollflow2/config_seviri_hrit
where mnt refers to the mount directory for production and config_seviri_hrit the folder where we store the configuration files. We can choose to have as many configuration directories as satellites we want to process data from, or instead create separate dockers for separate satellites.
As an example, we will consider what is inside the config_seviri_hrit folder. In there we should find the following information:
- areas.yaml
- environment.yaml
- segment_gatherer.yaml
- trollflow2_seviri_hrit.yaml
- supervisord.conf
- trollstalker.ini
- trollstalker_logging.ini
- /composites
- /enhancements
- /readers
- /writers
The areas.yaml and the directories can be obtained directly from the Satpy repository → Satpy Repository
Whilst the other files are available from the Trollflow2 repository → Trollflow2 Seviri HRIT Repository
Now we will get into each separate file and describe its main function.
This file contains a set of predefined areas to be used for resampling purposes. These areas can be altered if needed and new ones can be also created. They represent the region of interest when generating satellite images. In the trollflow2_seviri_hrit.yaml, when we select an area, it will be searched for inside this file.
This file contains a list of all the Python packages that are a prerequisite for the full processing. It should look something like:
In this file it is defined the name pattern of the satellite files, as well as what files are critical (for example: EPI & PRO files). This file is specific for each kind of satellite.
This file is the most important one because in there we can define exactly what composites and datasets will be generated and for how many different regions.
CAUTION! YAML files are VERY specific about indentation, so even the slightest displacement can translate into an error on the process. An extensive guide on how to write this type of files can be found here → YAML files Configuration
WARNING! If you would like to decorate the image with a logo, outline of the coasts or a piece of text, it is necessary to first install pydecorate on the python-conda environment of the Docker. To do so, we can access the terminal of the docker through the command docker exec -it trollflow2 /bin/bash and then enter conda install pydecorate or pip install pydecorate.
An example of an operative file is as follows:
The Supervisord website is made of a .conf file that contains all the specifications of the production chain, so this file is more or less the 'brain' of Trollflow2. The structure is composed of the relative paths and constants that the Pytroll scripts are summoning on. The important steps to change are the http_server specifications and the serverurl extension.
The main program settings are listed here and can be altered as needed:
[program:msg_0deg_trollflow2]
command = /opt/conda/bin/python /opt/conda/bin/satpy_launcher.py /mnt/config/trollflow2_seviri_hrit.yaml
redirect_stderr = true
stdout_logfile = /mnt/logs/supervisor_trollflow2_seviri_hrit.log
stopsignal = KILL
environment = DASK_NUM_WORKERS="1",OMP_NUM_THREADS="1",XRIT_DECOMPRESS_PATH="/opt/conda/bin/xRITDecompress"
For example, the /mnt/config folder is defined in the installation process and under the preferable path. WARNING! In case of a change of path, all other files in the configuration repository should also be checked and adjusted for the new path.
These files contain the parameters needed for the correct reception of the satellite files, such as the input folder path, the file pattern and platform name of the instrument. The logging SHOULD NOT BE CHANGED as it formats the debug messages and is generic for any type of instrument or satellite.
This directory and following 3 can be found inside the Satpy repository. This one contains all the .yaml composites files for each different satellite. Each file has a set of requisites for the different composites we can define. The full tutorial on how to generate an enhancement can be found here → Trollflow2 Composites & Enhancements Tutorial
This next directory contains the files related to the enhancements (e.g. colourized images). The first initial thought would be to consider it being the same as the composite one. However, the most important file in this case is the generic.yaml, which contains the enhancement requisites that are common for all satellites. This file is the first by priority that will be searched for and implemented in the chain of image generation. The generic file looks something like this:
This directory contains the readers, which are the files that specify the default channels each different satellite uses, as well as file patterns, wavelengths and resolution of each channel.
This is an example of the 0.6 Visible Channel extracted from seviri_l1b_nc.yaml.
VIS006:
name: VIS006
resolution: 3000.403165817
wavelength: [0.56, 0.635, 0.71]
calibration:
reflectance:
standard_name: toa_bidirectional_reflectance
units: "%"
radiance:
standard_name: toa_outgoing_radiance_per_unit_wavenumber
units: mW m-2 sr-1 (cm-1)-1
counts:
standard_name: counts
units: count
file_type: seviri_l1b_nc
nc_key: 'ch1'
This directory contains the writers, which are the files that specify how the writing (generation) of the images is done. At the moment there are 7 different writers, that have been created specifically for the different types of image extension you can set with Pytroll.
- cf
- geotiff
- simple image (png)
- ninjo tiff
- ninjo geotiff
- mi tiff
- awips tiled
Tutorial by @martabalague99