diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index b3ddad0307..47633c0c02 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -26,7 +26,7 @@ If applicable, add screenshots to help explain your problem. **Environment Info:** - OS: [e.g. OSX, Windows, Linux] - - SatPy Version: [e.g. 0.9.0] + - Satpy Version: [e.g. 0.9.0] - PyResample Version: - Readers and writers dependencies (when relevant): [run `from satpy.config import check_satpy; check_satpy()`] diff --git a/.gitignore b/.gitignore index adde65e36a..8e81eced92 100644 --- a/.gitignore +++ b/.gitignore @@ -63,3 +63,4 @@ nosetests.xml # vi / vim swp files *.swp +.DS_STORE diff --git a/AUTHORS.md b/AUTHORS.md index f2a0363d32..f441a55c32 100644 --- a/AUTHORS.md +++ b/AUTHORS.md @@ -33,3 +33,5 @@ The following people have made contributions to this project: - [sjoro (sjoro)](https://github.com/sjoro) - Guido della Bruna - meteoswiss - Marco Sassi - meteoswiss +- [Rohan Daruwala (rdaruwala)](https://github.com/rdaruwala) +- [Simon R. Proud (simonrp84)](https://github.com/simonrp84) diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 3e908d7977..0d030bfa09 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -2,15 +2,15 @@ How to contribute ================= -Thank you for considering contributing to SatPy! SatPy's development team +Thank you for considering contributing to Satpy! Satpy's development team is made up of volunteers so any help we can get is very appreciated. Contributions from users are what keep this community going. We welcome any contributions including bug reports, documentation fixes or updates, -bug fixes, and feature requests. By contributing to SatPy you are providing +bug fixes, and feature requests. By contributing to Satpy you are providing code that everyone can use and benefit from. -The following guidelines will describe how the SatPy project structures +The following guidelines will describe how the Satpy project structures its code contributions from discussion to code to package release. For more information on contributing to open source projects see @@ -39,7 +39,7 @@ What can I do? - Read the :doc:`index` for more details on contributing code. - `Fork `_ the repository on GitHub and install the package in development mode. -- Update the SatPy documentation to make it clearer and more detailed. +- Update the Satpy documentation to make it clearer and more detailed. - Contribute code to either fix a bug or add functionality and submit a `Pull Request `_. - Make an example Jupyter Notebook and add it to the @@ -53,25 +53,25 @@ fault. When you submit your changes to be merged as a GitHub `Pull Request `_ they will be automatically tested and checked against coding style rules. Before they are merged they are reviewed by at least one maintainer of the -SatPy project. If anything needs updating, we'll let you know. +Satpy project. If anything needs updating, we'll let you know. What is expected? ================= -You can expect the SatPy maintainers to help you. We are all volunteers, +You can expect the Satpy maintainers to help you. We are all volunteers, have jobs, and occasionally go on vacations. We will try our best to answer your questions as soon as possible. We will try our best to understand your use case and add the features you need. Although we strive to make -SatPy useful for everyone there may be some feature requests that we can't +Satpy useful for everyone there may be some feature requests that we can't allow if they would require breaking existing features. Other features may be best for a different package, PyTroll or otherwise. Regardless, we will help you find the best place for your feature and to make it possible to do what you want. -We, the SatPy maintainers, expect you to be patient, understanding, and -respectful of both developers and users. SatPy can only be successful if +We, the Satpy maintainers, expect you to be patient, understanding, and +respectful of both developers and users. Satpy can only be successful if everyone in the community feels welcome. We also expect you to put in as -much work as you expect out of us. There is no dedicated PyTroll or SatPy +much work as you expect out of us. There is no dedicated PyTroll or Satpy support team, so there may be times when you need to do most of the work to solve your problem (trying different test cases, environments, etc). @@ -79,7 +79,7 @@ Being respectful includes following the style of the existing code for any code submissions. Please follow `PEP8 `_ style guidelines and limit lines of code to 80 characters whenever possible and when it doesn't -hurt readability. SatPy follows +hurt readability. Satpy follows `Google Style Docstrings `_ for all code API documentation. When in doubt use the existing code as a guide for how coding should be done. @@ -89,7 +89,7 @@ guide for how coding should be done. How do I get help? ================== -The SatPy developers (and all other PyTroll package developers) monitor the: +The Satpy developers (and all other PyTroll package developers) monitor the: - `Mailing List `_ - `Slack chat `_ (get an `invitation `_) @@ -99,17 +99,17 @@ How do I submit my changes? =========================== Any contributions should start with some form of communication (see above) to -let the SatPy maintainers know how you plan to help. The larger the +let the Satpy maintainers know how you plan to help. The larger the contribution the more important direct communication is so everyone can avoid duplicate code and wasted time. -After talking to the SatPy developers any additional work like code or +After talking to the Satpy developers any additional work like code or documentation changes can be provided as a GitHub `Pull Request `_. Code of Conduct =============== -SatPy follows the same code of conduct as the PyTroll project. For reference +Satpy follows the same code of conduct as the PyTroll project. For reference it is copied to this repository in `CODE_OF_CONDUCT.md `_. diff --git a/README.rst b/README.rst index cd0c2ea56f..424bf544d1 100644 --- a/README.rst +++ b/README.rst @@ -14,9 +14,9 @@ Satpy :target: https://badge.fury.io/py/satpy -The SatPy package is a python library for reading and manipulating +The Satpy package is a python library for reading and manipulating meteorological remote sensing data and writing it to various image and -data file formats. SatPy comes with the ability to make various RGB +data file formats. Satpy comes with the ability to make various RGB composites directly from satellite instrument channel data or higher level processing output. The `pyresample `_ package is used @@ -28,7 +28,7 @@ http://satpy.readthedocs.org/. Installation ------------ -SatPy can be installed from PyPI with pip: +Satpy can be installed from PyPI with pip: .. code-block:: bash @@ -44,7 +44,7 @@ It is also available from `conda-forge` for conda installations: Code of Conduct --------------- -SatPy follows the same code of conduct as the PyTroll project. For reference +Satpy follows the same code of conduct as the PyTroll project. For reference it is copied to this repository in CODE_OF_CONDUCT.md_. As stated in the PyTroll home page, this code of conduct applies to the diff --git a/RELEASING.md b/RELEASING.md index d9f73483b6..8b70410716 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,24 +1,36 @@ -# Releasing SatPy +# Releasing Satpy 1. checkout master 2. pull from repo 3. run the unittests -4. run `loghub` and update the `CHANGELOG.md` file: +4. run `loghub`. Replace and with proper + values. To get the previous version run `git tag` and select the most + recent with highest version number. ``` -loghub pytroll/satpy -u -st v0.8.0 -plg bug "Bugs fixed" -plg enhancement "Features added" -plg documentation "Documentation changes" -plg backwards-incompatibility "Backwards incompatible changes" +loghub pytroll/satpy -u -st v -plg bug "Bugs fixed" -plg enhancement "Features added" -plg documentation "Documentation changes" -plg backwards-incompatibility "Backwards incompatible changes" ``` -Don't forget to commit! +This command will create a CHANGELOG.temp file which need to be added +to the top of the CHANGLOG.md file. The same content is also printed +to terminal, so that can be copy-pasted, too. Remember to update also +the version number to the same given in step 5. Don't forget to commit +CHANGELOG.md! 5. Create a tag with the new version number, starting with a 'v', eg: ``` -git tag -a v0.22.45 -m "Version 0.22.45" +git tag -a v -m "Version " ``` -See [semver.org](http://semver.org/) on how to write a version number. +For example if the previous tag was `v0.9.0` and the new release is a +patch release, do: +``` +git tag -a v0.9.1 -m "Version 0.9.1" +``` + +See [semver.org](http://semver.org/) on how to write a version number. 6. push changes to github `git push --follow-tags` diff --git a/doc/README b/doc/README index 6ce416d517..15ee4b9949 100644 --- a/doc/README +++ b/doc/README @@ -4,7 +4,7 @@ by running: make html The generated HTML documentation pages are available in `build/html`. If -SatPy's API has changed (new functions, modules, classes, etc) then the +Satpy's API has changed (new functions, modules, classes, etc) then the API documentation should be regenerated before running the above make command. diff --git a/doc/source/api/satpy.demo.rst b/doc/source/api/satpy.demo.rst new file mode 100644 index 0000000000..f06476c4d1 --- /dev/null +++ b/doc/source/api/satpy.demo.rst @@ -0,0 +1,22 @@ +satpy.demo package +================== + +Submodules +---------- + +satpy.demo.google\_cloud\_platform module +----------------------------------------- + +.. automodule:: satpy.demo.google_cloud_platform + :members: + :undoc-members: + :show-inheritance: + + +Module contents +--------------- + +.. automodule:: satpy.demo + :members: + :undoc-members: + :show-inheritance: diff --git a/doc/source/api/satpy.readers.rst b/doc/source/api/satpy.readers.rst index 7a93368d76..669f7118a2 100644 --- a/doc/source/api/satpy.readers.rst +++ b/doc/source/api/satpy.readers.rst @@ -364,6 +364,14 @@ satpy.readers.viirs\_compact module :undoc-members: :show-inheritance: +satpy.readers.viirs\_edr\_active\_fires module +-------------------------------------- + +.. automodule:: satpy.readers.viirs_edr_active_fires + :members: + :undoc-members: + :show-inheritance: + satpy.readers.viirs\_edr\_flood module -------------------------------------- diff --git a/doc/source/api/satpy.rst b/doc/source/api/satpy.rst index 4829695c03..a0e7f79db3 100644 --- a/doc/source/api/satpy.rst +++ b/doc/source/api/satpy.rst @@ -7,6 +7,7 @@ Subpackages .. toctree:: satpy.composites + satpy.demo satpy.enhancements satpy.readers satpy.writers diff --git a/doc/source/composites.rst b/doc/source/composites.rst index 576905335a..f3f41a513f 100644 --- a/doc/source/composites.rst +++ b/doc/source/composites.rst @@ -2,26 +2,46 @@ Composites ========== -Documentation coming soon... +Built-in Compositors +==================== -Modifiers -========= +.. py:currentmodule:: satpy.composites -Making custom composites -======================== +There are several built-in compositors available in SatPy. +All of them use the :class:`GenericCompositor` base class +which handles various image modes (`L`, `LA`, `RGB`, and +`RGBA` at the moment) and updates attributes. -.. note:: +The below sections summarize the composites that come with SatPy and +show basic examples of creating and using them with an existing +:class:`~satpy.scene.Scene` object. It is recommended that any composites +that are used repeatedly be configured in YAML configuration files. +General-use compositor code dealing with visible or infrared satellite +data can be put in a configuration file called ``visir.yaml``. Composites +that are specific to an instrument can be placed in YAML config files named +accordingly (e.g., ``seviri.yaml`` or ``viirs.yaml``). See the +`satpy repository `_ +for more examples. - These features will be added to the ``Scene`` object in the future. +GenericCompositor +----------------- -Building custom composites makes use of the :class:`GenericCompositor` class. For example, -building an overview composite can be done manually with:: +:class:`GenericCompositor` class can be used to create basic single +channel and RGB composites. For example, building an overview composite +can be done manually within Python code with:: >>> from satpy.composites import GenericCompositor - >>> compositor = GenericCompositor("myoverview", "bla", "") + >>> compositor = GenericCompositor("overview") >>> composite = compositor([local_scene[0.6], ... local_scene[0.8], ... local_scene[10.8]]) + +One important thing to notice is that there is an internal difference +between a composite and an image. A composite is defined as a special +dataset which may have several bands (like `R`, `G` and `B` bands). However, +the data isn't stretched, or clipped or gamma filtered until an image +is generated. To get an image out of the above composite:: + >>> from satpy.writers import to_image >>> img = to_image(composite) >>> img.invert([False, False, True]) @@ -29,21 +49,357 @@ building an overview composite can be done manually with:: >>> img.gamma(1.7) >>> img.show() +This part is called `enhancement`, and is covered in more detail in +:doc:`enhancements`. + + +DifferenceCompositor +-------------------- + +:class:`DifferenceCompositor` calculates a difference of two datasets:: + + >>> from satpy.composites import DifferenceCompositor + >>> compositor = DifferenceCompositor("diffcomp") + >>> composite = compositor([local_scene[10.8], local_scene[12.0]]) + +FillingCompositor +----------------- + +:class:`FillingCompositor`:: fills the missing values in three datasets +with the values of another dataset::: + + >>> from satpy.composites import FillingCompositor + >>> compositor = FillingCompositor("fillcomp") + >>> filler = local_scene[0.6] + >>> data_with_holes_1 = local_scene['ch_a'] + >>> data_with_holes_2 = local_scene['ch_b'] + >>> data_with_holes_3 = local_scene['ch_c'] + >>> composite = compositor([filler, data_with_holes_1, data_with_holes_2, + ... data_with_holes_3]) + +PaletteCompositor +------------------ + +:class:`PaletteCompositor` creates a color version of a single channel +categorical dataset using a colormap:: + + >>> from satpy.composites import PaletteCompositor + >>> compositor = PaletteCompositor("palcomp") + >>> composite = compositor([local_scene['cma'], local_scene['cma_pal']]) + +The palette should have a single entry for all the (possible) values +in the dataset mapping the value to an RGB triplet. Typically the +palette comes with the categorical (e.g. cloud mask) product that is +being visualized. + +DayNightCompositor +------------------ + +:class:`DayNightCompositor` merges two different composites. The +first composite will be placed on the day-side of the scene, and the +second one on the night side. The transition from day to night is +done by calculating solar zenith angle (SZA) weighed average of the +two composites. The SZA can optionally be given as third dataset, and +if not given, the angles will be calculated. Width of the blending +zone can be defined when initializing the compositor (default values +shown in the example below). + + >>> from satpy.composites import DayNightCompositor + >>> compositor = DayNightCompositor("dnc", lim_low=85., lim_high=95.) + >>> composite = compositor([local_scene['true_color'], + ... local_scene['night_fog']]) + +RealisticColors +--------------- + +:class:`RealisticColors` compositor is a special compositor that is +used to create realistic near-true-color composite from MSG/SEVIRI +data:: + + >>> from satpy.composites import RealisticColors + >>> compositor = RealisticColors("realcols", lim_low=85., lim_high=95.) + >>> composite = compositor([local_scene['VIS006'], + ... local_scene['VIS008'], + ... local_scene['HRV']]) + +CloudCompositor +--------------- + +:class:`CloudCompositor` can be used to threshold the data so that +"only" clouds are visible. These composites can be used as an overlay +on top of e.g. static terrain images to show a rough idea where there +are clouds. The data are thresholded using three variables:: -One important thing to notice is that there is an internal difference between a composite and an image. A composite -is defined as a special dataset which may have several bands (like R, G, B bands). However, the data isn't stretched, -or clipped or gamma filtered until an image is generated. + - `transition_min`: values below or equal to this are clouds -> opaque white + - `transition_max`: values above this are cloud free -> transparent + - `transition_gamma`: gamma correction applied to clarify the clouds +Usage (with default values):: + + >>> from satpy.composites import CloudCompositor + >>> compositor = CloudCompositor("clouds", transition_min=258.15, + ... transition_max=298.15, + ... transition_gamma=3.0) + >>> composite = compositor([local_scene[10.8]]) + +Support for using this compositor for VIS data, where the values for +high/thick clouds tend to be in reverse order to brightness +temperatures, is to be added. + +RatioSharpenedRGB +----------------- + +:class:`RatioSharpenedRGB` + +SelfSharpenedRGB +---------------- + +:class:`SelfSharpenedRGB` sharpens the RGB with ratio of a band with a +strided version of itself. + +LuminanceSharpeningCompositor +----------------------------- + +:class:`LuminanceSharpeningCompositor` replaces the luminance from an +RGB composite with luminance created from reflectance data. If the +resolutions of the reflectance data _and_ of the target area +definition are higher than the base RGB, more details can be +retrieved. This compositor can be useful also with matching +resolutions, e.g. to highlight shadowing at cloudtops in colorized +infrared composite. + + >>> from satpy.composites import LuminanceSharpeningCompositor + >>> compositor = LuminanceSharpeningCompositor("vis_sharpened_ir") + >>> vis_data = local_scene['HRV'] + >>> colorized_ir_clouds = local_scene['colorized_ir_clouds'] + >>> composite = compositor([vis_data, colorized_ir_clouds]) + +SandwichCompositor +------------------ + +Similar to :class:`LuminanceSharpeningCompositor`, +:class:`SandwichCompositor` uses reflectance data to bring out more +details out of infrared or low-resolution composites. +:class:`SandwichCompositor` multiplies the RGB channels with (scaled) +reflectance. + + >>> from satpy.composites import SandwichCompositor + >>> compositor = SandwichCompositor("ir_sandwich") + >>> vis_data = local_scene['HRV'] + >>> colorized_ir_clouds = local_scene['colorized_ir_clouds'] + >>> composite = compositor([vis_data, colorized_ir_clouds]) + +Creating composite configuration files +====================================== To save the custom composite, the following procedure can be used: 1. Create a custom directory for your custom configs. -2. Set it in the environment variable called PPP_CONFIG_DIR. -3. Write config files with your changes only (look at eg satpy/etc/composites/seviri.yaml for inspiration), pointing to the custom module containing your composites. Don't forget to add changes to the enhancement/generic.cfg file too. -4. Put your composites module on the python path. +2. Set the environment variable ``PPP_CONFIG_DIR`` to this path. +3. Write config files with your changes only (see examples below), pointing + to the (custom) module containing your composites. Generic compositors can + be placed in ``$PPP_CONFIG_DIR/composites/visir.yaml`` and instrument- + specific ones in ``$PPP_CONFIG_DIR/composites/.yaml``. Don't forget + to add changes to the ``enhancement/generic.yaml`` file too. +4. If custom compositing code was used then it must be importable by python. + If the code is not installed in your python environment then another option + it to add it to your ``PYTHONPATH``. + +With that, you should be able to load your new composite directly. Example +configuration files can be found in the satpy repository as well as a few +simple examples below. + +Simple RGB composite +-------------------- + +This is the overview composite shown in the first code example above +using :class:`GenericCompositor`:: + + sensor_name: visir + + composites: + overview: + compositor: !!python/name:satpy.composites.GenericCompositor + prerequisites: + - 0.6 + - 0.8 + - 10.8 + standard_name: overview + +For an instrument specific version (here MSG/SEVIRI), we should use +the channel _names_ instead of wavelengths. Note also that the +sensor_name is now combination of visir and seviri, which means that +it extends the generic visir composites:: + + sensor_name: visir/seviri -With that, you should be able to load your new composite directly. + composites: + + overview: + compositor: !!python/name:satpy.composites.GenericCompositor + prerequisites: + - VIS006 + - VIS008 + - IR_108 + standard_name: overview + +In the following examples only the composite receipes are shown, and +the header information (sensor_name, composites) and intendation needs +to be added. + +Using modifiers +--------------- + +In many cases the basic datasets need to be adjusted, e.g. for Solar +zenith angle normalization. These modifiers can be applied in the +following way:: + + overview: + compositor: !!python/name:satpy.composites.GenericCompositor + prerequisites: + - name: VIS006 + modifiers: [sunz_corrected] + - name: VIS008 + modifiers: [sunz_corrected] + - IR_108 + standard_name: overview + +Here we see two changes: + +1. channels with modifiers need to have either `name` or `wavelength` + added in front of the channel name or wavelength, respectively +2. a list of modifiers attached to the dictionary defining the channel + +The modifier above is a built-in that normalizes the Solar zenith +angle to Sun being directly at the zenith. + +Using other composites +---------------------- + +Often it is handy to use other composites as a part of the composite. +In this example we have one composite that relies on solar channels on +the day side, and another for the night side:: + + natural_with_night_fog: + compositor: !!python/name:satpy.composites.DayNightCompositor + prerequisites: + - natural_color + - night_fog + standard_name: natural_with_night_fog + +This compositor has two additional keyword arguments that can be +defined (shown with the default values, thus identical result as +above):: + + natural_with_night_fog: + compositor: !!python/name:satpy.composites.DayNightCompositor + prerequisites: + - natural_color + - night_fog + lim_low: 85.0 + lim_high: 95.0 + standard_name: natural_with_night_fog + +Defining other composites in-line +--------------------------------- + +It is also possible to define sub-composites in-line. This example is +the built-in airmass composite:: + + airmass: + compositor: !!python/name:satpy.composites.GenericCompositor + prerequisites: + - compositor: !!python/name:satpy.composites.DifferenceCompositor + prerequisites: + - wavelength: 6.2 + - wavelength: 7.3 + - compositor: !!python/name:satpy.composites.DifferenceCompositor + prerequisites: + - wavelength: 9.7 + - wavelength: 10.8 + - wavelength: 6.2 + standard_name: airmass + +Enhancing the images +==================== .. todo:: - How to save custom-made composites + Explain how composite names, composite standard_name, enhancement + names, and enhancement standard_name are related to each other + + Explain what happens when no enhancement is configured for a + product (= use the default enhancement). + + Explain that the methods are often just a wrapper for XRImage + methods, but can also be something completely custom. + + List and explain in detail the built-in enhancements: + + - stretch + - gamma + - invert + - crefl_scaling + - cira_stretch + - lookup + - colorize + - palettize + - three_d_effect + - btemp_threshold + +.. todo:: + + Should this be in another file/page? + +After the composite is defined and created, it needs to be converted +to an image. To do this, it is necessary to describe how the data +values are mapped to values stored in the image format. This +procedure is called ``stretching``, and in SatPy it is implemented by +``enhancements``. + +The first step is to convert the composite to an +:class:`~trollimage.xrimage.XRImage` object:: + + >>> from satpy.writers import to_image + >>> img = to_image(composite) + +Now it is possible to apply enhancements available in the class:: + + >>> img.invert([False, False, True]) + >>> img.stretch("linear") + >>> img.gamma(1.7) + +And finally either show or save the image:: + + >>> img.show() + >>> img.save('image.tif') + +As pointed out in the composite section, it is better to define +frequently used enhancements in configuration files under +``$PPP_CONFIG_DIR/enhancements/``. The enhancements can either be in +``generic.yaml`` or instrument-specific file (e.g., ``seviri.yaml``). + +The above enhancement can be written (with the headers necessary for +the file) as:: + + enhancements: + overview: + standard_name: overview + operations: + - name: inverse + method: !!python/name:satpy.enhancements.invert + args: [False, False, True] + - name: stretch + method: !!python/name:satpy.enhancements.stretch + kwargs: + stretch: linear + - name: gamma + method: !!python/name:satpy.enhancements.gamma + kwargs: + gamma: [1.7, 1.7, 1.7] + +More examples can be found in SatPy source code directory +``satpy/etc/enhancements/generic.yaml``. + +See the :doc:`enhancements` documentation for more information on +available built-in enhancements. diff --git a/doc/source/conf.py b/doc/source/conf.py index eb0dbec0bb..0f6ce1e4a4 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -60,7 +60,7 @@ def __getattr__(cls, name): # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx.ext.coverage', - 'sphinx.ext.doctest', 'sphinx.ext.napoleon', 'doi_role'] + 'sphinx.ext.doctest', 'sphinx.ext.napoleon', 'sphinx.ext.autosummary', 'doi_role'] # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] @@ -75,7 +75,7 @@ def __getattr__(cls, name): master_doc = 'index' # General information about the project. -project = u'SatPy' +project = u'Satpy' copyright = u'2009-{}, The PyTroll Team'.format(datetime.utcnow().strftime("%Y")) # The version info for the project you're documenting, acts as replacement for diff --git a/doc/source/data_download.rst b/doc/source/data_download.rst index d6dc9b7158..e6e1a38017 100644 --- a/doc/source/data_download.rst +++ b/doc/source/data_download.rst @@ -1,9 +1,9 @@ Downloading Data ================ -One of the main features of SatPy is its ability to read various satellite +One of the main features of Satpy is its ability to read various satellite data formats. However, it does not currently provide any functionality for -downloading data from any remote sources. SatPy assumes all data is available +downloading data from any remote sources. Satpy assumes all data is available through the local system, either as a local directory or network mounted file systems. Certain readers that use ``xarray`` to open data files may be able to load files from remote systems by using OpenDAP or similar @@ -22,7 +22,7 @@ satellite mission organization (NOAA, NASA, EUMETSAT, etc). In these cases data is usually available as a mounted network file system and can be accessed like a normal local path (with the added latency of network communications). -Below are some data sources that provide data that can be read by SatPy. If +Below are some data sources that provide data that can be read by Satpy. If you know of others please let us know by either creating a GitHub issue or pull request. diff --git a/doc/source/dev_guide/custom_reader.rst b/doc/source/dev_guide/custom_reader.rst index 5310fa68a0..8e4c491bd5 100644 --- a/doc/source/dev_guide/custom_reader.rst +++ b/doc/source/dev_guide/custom_reader.rst @@ -1,5 +1,5 @@ ================================= - Adding a Custom Reader to SatPy + Adding a Custom Reader to Satpy ================================= In order to add a reader to satpy, you will need to create two files: @@ -14,7 +14,7 @@ format for SEVIRI data Naming your reader ------------------ -SatPy tries to follow a standard scheme for naming its readers. These names +Satpy tries to follow a standard scheme for naming its readers. These names are used in filenames, but are also used by users so it is important that the name be recognizable and clear. Although some special cases exist, most fit in to the following naming scheme: @@ -57,7 +57,7 @@ if needed (ex. goes-imager). The existing :ref:`reader's table ` can be used for reference. When in doubt, reader names can be discussed in the github pull -request when this reader is added to SatPy or a github issue. +request when this reader is added to Satpy or a github issue. The YAML file ------------- diff --git a/doc/source/dev_guide/index.rst b/doc/source/dev_guide/index.rst index dbff0044d3..608b1a0e95 100644 --- a/doc/source/dev_guide/index.rst +++ b/doc/source/dev_guide/index.rst @@ -18,19 +18,19 @@ at the pages listed below. Coding guidelines ================= -SatPy is part of `PyTroll `_, +Satpy is part of `PyTroll `_, and all code should follow the `PyTroll coding guidelines and best practices `_. -SatPy currently supports Python 2.7 and 3.4+. All code should be written to +Satpy currently supports Python 2.7 and 3.4+. All code should be written to be compatible with these versions. Development installation ======================== See the :doc:`../install` section for basic installation instructions. When -it comes time to install SatPy it should be installed from a clone of the git +it comes time to install Satpy it should be installed from a clone of the git repository and in development mode so that local file changes are automatically reflected in the python environment. We highly recommend making a separate conda environment or virtualenv for development. @@ -44,7 +44,7 @@ clone your fork. The package can then be installed in development by doing:: Running tests ============= -SatPy tests are written using the python :mod:`unittest` module and the tests +Satpy tests are written using the python :mod:`unittest` module and the tests can be executed by running:: python setup.py test @@ -52,7 +52,7 @@ can be executed by running:: Documentation ============= -SatPy's documentation is built using Sphinx. All documentation lives in the +Satpy's documentation is built using Sphinx. All documentation lives in the ``doc/`` directory of the project repository. After editing the source files there the documentation can be generated locally:: diff --git a/doc/source/dev_guide/xarray_migration.rst b/doc/source/dev_guide/xarray_migration.rst index 4f8d27d340..4d19c4526d 100644 --- a/doc/source/dev_guide/xarray_migration.rst +++ b/doc/source/dev_guide/xarray_migration.rst @@ -5,16 +5,16 @@ Migrating to xarray and dask Many python developers dealing with meteorologic satellite data begin with using NumPy arrays directly. This work usually involves masked arrays, boolean masks, index arrays, and reshaping. Due to the libraries used by -SatPy these operations can't always be done in the same way. This guide acts -as a starting point for new SatPy developers in transitioning from NumPy's -array operations to SatPy's operations, although they are very similar. +Satpy these operations can't always be done in the same way. This guide acts +as a starting point for new Satpy developers in transitioning from NumPy's +array operations to Satpy's operations, although they are very similar. To provide the most functionality for users, -SatPy uses the `xarray `_ library's +Satpy uses the `xarray `_ library's :class:`~xarray.DataArray` object as the main representation for its data. DataArray objects can also benefit from the `dask `_ library. The combination of -these libraries allow SatPy to easily distribute operations over multiple +these libraries allow Satpy to easily distribute operations over multiple workers, lazy evaluate operations, and keep track additional metadata and coordinate information. @@ -38,7 +38,7 @@ To create such an array, you can do for example attrs={'sensor': 'olci'}) where ``my_data`` can be a regular numpy array, a numpy memmap, or, if you -want to keep things lazy, a dask array (more on dask later). SatPy uses dask +want to keep things lazy, a dask array (more on dask later). Satpy uses dask arrays with all of its DataArrays. Dimensions @@ -235,7 +235,7 @@ Regular arithmetic operations are provided, and generate another dask array. In order to compute the actual data during testing, use the :func:`~dask.compute` method. -In normal SatPy operations you will want the data to be evaluated as late as +In normal Satpy operations you will want the data to be evaluated as late as possible to improve performance so `compute` should only be used when needed. >>> (arr1 + arr2).compute() diff --git a/doc/source/enhancements.rst b/doc/source/enhancements.rst new file mode 100644 index 0000000000..44e1c0f9fc --- /dev/null +++ b/doc/source/enhancements.rst @@ -0,0 +1,91 @@ +============ +Enhancements +============ + +Built-in enhancement methods +============================ + +stretch +------- + +The most basic operation is to stretch the image so that the data fits +to the output format. There are many different ways to stretch the +data, which are configured by giving them in `kwargs` dictionary, like +in the example above. The default, if nothing else is defined, is to +apply a linear stretch. For more details, see below. + +linear +****** + +As the name suggests, linear stretch converts the input values to +output values in a linear fashion. By default, 5% of the data is cut +on both ends of the scale, but these can be overridden with +``cutoffs=(0.005, 0.005)`` argument:: + + - name: stretch + method: !!python/name:satpy.enhancements.stretch + kwargs: + stretch: linear + cutoffs: (0.003, 0.005) + +.. note:: + + This enhancement is currently not optimized for dask because it requires + getting minimum/maximum information for the entire data array. + +crude +***** + +The crude stretching is used to limit the input values to a certain +range by clipping the data. This is followed by a linear stretch with +no cutoffs specified (see above). Example:: + + - name: stretch + method: !!python/name:satpy.enhancements.stretch + kwargs: + stretch: crude + min_stretch: [0, 0, 0] + max_stretch: [100, 100, 100] + +It is worth noting that this stretch can also be used to _invert_ the +data by giving larger values to the min_stretch than to max_stretch. + +histogram +********* + +gamma +----- + +invert +------ + +crefl_scaling +------------- + +cira_stretch +------------ + +lookup +------ + +colorize +-------- + +palettize +--------- + +three_d_effect +-------------- + +The `three_d_effect` enhancement adds an 3D look to an image by +convolving with a 3x3 kernel. User can adjust the strength of the +effect by determining the weight (default: 1.0). Example:: + + - name: 3d_effect + method: !!python/name:satpy.enhancements.three_d_effect + kwargs: + weight: 1.0 + + +btemp_threshold +--------------- diff --git a/doc/source/examples.rst b/doc/source/examples.rst index 43b8d9f74c..5ef7c98875 100644 --- a/doc/source/examples.rst +++ b/doc/source/examples.rst @@ -1,7 +1,7 @@ Examples ======== -SatPy examples are available as Jupyter Notebooks on the +Satpy examples are available as Jupyter Notebooks on the `pytroll-examples `_ git repository. They include python code, PNG images, and descriptions of what the example is doing. Below is a list of some of the examples and a brief @@ -22,7 +22,7 @@ as explanations in the various sections of this documentation. * - `Sentinel-3 OLCI True Color `_ - Reading OLCI data from Sentinel 3 with Pytroll/Satpy * - `Sentinel 2 MSI true color `_ - - Reading MSI data from Sentinel 2 with Pytroll/SatPy + - Reading MSI data from Sentinel 2 with Pytroll/Satpy * - `Suomi-NPP VIIRS SDR True Color `_ - Generate a rayleigh corrected true color RGB from VIIRS I- and M-bands * - `Aqua/Terra MODIS True Color `_ diff --git a/doc/source/faq.rst b/doc/source/faq.rst index 857cd55453..b9b08eadd1 100644 --- a/doc/source/faq.rst +++ b/doc/source/faq.rst @@ -2,7 +2,7 @@ FAQ === Below you'll find frequently asked questions, performance tips, and other -topics that don't really fit in to the rest of the SatPy documentation. +topics that don't really fit in to the rest of the Satpy documentation. If you have any other questions that aren't answered here feel free to make an issue on GitHub or talk to us on the Slack team or mailing list. See the @@ -12,16 +12,16 @@ an issue on GitHub or talk to us on the Slack team or mailing list. See the :depth: 1 :local: -Why is SatPy slow on my powerful machine? +Why is Satpy slow on my powerful machine? ----------------------------------------- -SatPy depends heavily on the dask library for its performance. However, +Satpy depends heavily on the dask library for its performance. However, on some systems dask's default settings can actually hurt performance. By default dask will create a "worker" for each logical core on your system. In most systems you have twice as many logical cores (also known as threaded cores) as physical cores. Managing and communicating with all of these workers can slow down dask, especially when they aren't all -being used by most SatPy calculations. One option is to limit the number of +being used by most Satpy calculations. One option is to limit the number of workers by doing the following at the **top** of your python code: .. code-block:: python @@ -29,7 +29,7 @@ workers by doing the following at the **top** of your python code: import dask from multiprocessing.pool import ThreadPool dask.config.set(pool=ThreadPool(8)) - # all other SatPy imports and code + # all other Satpy imports and code This will limit dask to using 8 workers. Typically numbers between 4 and 8 are good starting points. Number of workers can also be set from an @@ -43,7 +43,7 @@ isn't necessary: Similarly, if you have many workers processing large chunks of data you may be using much more memory than you expect. If you limit the number of workers *and* the size of the data chunks being processed by each worker you can -reduce the overall memory usage. Default chunk size can be configured in SatPy +reduce the overall memory usage. Default chunk size can be configured in Satpy by setting the following environment variable: .. code-block:: bash @@ -51,7 +51,7 @@ by setting the following environment variable: export PYTROLL_CHUNK_SIZE=2048 This could also be set inside python using ``os.environ``, but must be set -**before** SatPy is imported. This value defaults to 4096, meaning each +**before** Satpy is imported. This value defaults to 4096, meaning each chunk of data will be 4096 rows by 4096 columns. In the future setting this value will change to be easier to set in python. @@ -93,8 +93,8 @@ How do I avoid memory errors? ----------------------------- If your environment is using many dask workers, it may be using more memory -than it needs to be using. See the "Why is SatPy slow on my powerful machine?" -question above for more information on changing SatPy's memory usage. +than it needs to be using. See the "Why is Satpy slow on my powerful machine?" +question above for more information on changing Satpy's memory usage. Reducing GDAL output size? -------------------------- diff --git a/doc/source/index.rst b/doc/source/index.rst index 7084e4fc74..2784645072 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -1,16 +1,16 @@ ===================== -SatPy's Documentation +Satpy's Documentation ===================== -SatPy is a python library for reading and manipulating +Satpy is a python library for reading and manipulating meteorological remote sensing data and writing it to various image and -data file formats. SatPy comes with the ability to make various RGB +data file formats. Satpy comes with the ability to make various RGB composites directly from satellite instrument channel data or higher level processing output. The `pyresample `_ package is used to resample data to different uniform areas or grids. Various atmospheric corrections and visual enhancements are also provided, either directly in -SatPy or from those in the +Satpy or from those in the `PySpectral `_ and `TrollImage `_ packages. @@ -18,12 +18,12 @@ Go to the project_ page for source code and downloads. It is designed to be easily extendable to support any meteorological satellite by the creation of plugins (readers, compositors, writers, etc). The table at -the bottom of this page shows the input formats supported by the base SatPy +the bottom of this page shows the input formats supported by the base Satpy installation. .. note:: - SatPy's interfaces are not guaranteed stable and may change until version + Satpy's interfaces are not guaranteed stable and may change until version 1.0 when backwards compatibility will be a main focus. .. _project: http://github.com/pytroll/satpy @@ -39,6 +39,7 @@ installation. readers composites resample + enhancements writers multiscene dev_guide/index @@ -46,12 +47,12 @@ installation. .. toctree:: :maxdepth: 1 - SatPy API + Satpy API faq .. _reader_table: -.. list-table:: SatPy Readers +.. list-table:: Satpy Readers :header-rows: 1 :widths: 45 25 30 @@ -166,6 +167,9 @@ installation. * - AAPP MAIA VIIRS and AVHRR products in hdf5 format - `maia` - Nominal + * - VIIRS EDR Active Fires data in NetCDF4 & CSV .txt format + - `viirs_edr_active_fires` + - Beta * - VIIRS EDR Flood data in hdf4 format - `viirs_edr_flood` - Beta @@ -175,6 +179,9 @@ installation. * - SCMI ABI L1B format - `abi_l1b_scmi` - Beta + * - VIRR data in HDF5 format + - `virr_l1b` + - Beta Indices and tables ================== diff --git a/doc/source/install.rst b/doc/source/install.rst index 3042e41d19..63381890e0 100644 --- a/doc/source/install.rst +++ b/doc/source/install.rst @@ -5,7 +5,7 @@ Installation Instructions Pip-based Installation ====================== -SatPy is available from the Python Packaging Index (PyPI). A sandbox +Satpy is available from the Python Packaging Index (PyPI). A sandbox environment for `satpy` can be created using `Virtualenv `_. @@ -34,7 +34,7 @@ dependencies: Conda-based Installation ======================== -Starting with version 0.9, SatPy is available from the conda-forge channel. If +Starting with version 0.9, Satpy is available from the conda-forge channel. If you have not configured your conda environment to search conda-forge already then do: @@ -42,7 +42,7 @@ then do: $ conda config --add channels conda-forge -Then to install SatPy in to your current environment run: +Then to install Satpy in to your current environment run: .. code-block:: bash @@ -50,18 +50,18 @@ Then to install SatPy in to your current environment run: .. note:: - SatPy only automatically installs the dependencies needed to process the + Satpy only automatically installs the dependencies needed to process the most common use cases. Additional dependencies may need to be installed with conda or pip if import errors are encountered. Ubuntu System Python Installation ================================= -To install SatPy on an Ubuntu system we recommend using virtual environments -to separate SatPy and its dependencies from the rest of the system. Note that +To install Satpy on an Ubuntu system we recommend using virtual environments +to separate Satpy and its dependencies from the rest of the system. Note that these instructions require using "sudo" privileges which may not be available to all users and can be very dangerous. The following instructions attempt -to install some SatPy dependencies using the Ubuntu `apt` package manager to +to install some Satpy dependencies using the Ubuntu `apt` package manager to ease installation. Replace `/path/to/pytroll-env` with the environment to be created. diff --git a/doc/source/multiscene.rst b/doc/source/multiscene.rst index 534a4d5adf..0bb60c3bd1 100644 --- a/doc/source/multiscene.rst +++ b/doc/source/multiscene.rst @@ -1,11 +1,11 @@ MultiScene (Experimental) ========================= -Scene objects in SatPy are meant to represent a single geographic region at +Scene objects in Satpy are meant to represent a single geographic region at a specific single instant in time or range of time. This means they are not suited for handling multiple orbits of polar-orbiting satellite data, multiple time steps of geostationary satellite data, or other special data -cases. To handle these cases SatPy provides the `MultiScene` class. The below +cases. To handle these cases Satpy provides the `MultiScene` class. The below examples will walk through some basic use cases of the MultiScene. .. warning:: @@ -91,7 +91,7 @@ time. .. versionadded:: 0.12 - The ``from_files`` and ``group_files`` functions were added in SatPy 0.12. + The ``from_files`` and ``group_files`` functions were added in Satpy 0.12. See below for an alternative solution. This will compute one video frame (image) at a time and write it to the MPEG-4 @@ -103,7 +103,7 @@ for information on creating a ``Client`` object. If working on a cluster you may want to use :doc:`dask jobqueue ` to take advantage of multiple nodes at a time. -For older versions of SatPy we can manually create the `Scene` objects used. +For older versions of Satpy we can manually create the `Scene` objects used. The :func:`~glob.glob` function and for loops are used to group files into Scene objects that, if used individually, could load the data we want. The code below is equivalent to the ``from_files`` code above: diff --git a/doc/source/overview.rst b/doc/source/overview.rst index c0c4fbff02..d98e910dcf 100644 --- a/doc/source/overview.rst +++ b/doc/source/overview.rst @@ -2,21 +2,21 @@ Overview ======== -SatPy is designed to provide easy access to common operations for processing +Satpy is designed to provide easy access to common operations for processing meteorological remote sensing data. Any details needed to perform these -operations are configured internally to SatPy meaning users should not have to +operations are configured internally to Satpy meaning users should not have to worry about *how* something is done, only ask for what they want. Most of the -features provided by SatPy can be configured by keyword arguments (see the +features provided by Satpy can be configured by keyword arguments (see the :doc:`API Documentation ` or other specific section for more details). -For more complex customizations or added features SatPy uses a set of +For more complex customizations or added features Satpy uses a set of configuration files that can be modified by the user. The various components -and concepts of SatPy are described below. The :doc:`quickstart` guide also -provides simple example code for the available features of SatPy. +and concepts of Satpy are described below. The :doc:`quickstart` guide also +provides simple example code for the available features of Satpy. Scene ===== -SatPy provides most of its functionality through the +Satpy provides most of its functionality through the :class:`~satpy.scene.Scene` class. This acts as a container for the datasets being operated on and provides methods for acting on those datasets. It attempts to reduce the amount of low-level knowledge needed by the user while @@ -30,9 +30,9 @@ it is not guaranteed that all functionality works in these situations. DataArrays ========== -SatPy's lower-level container for data is the +Satpy's lower-level container for data is the :class:`xarray.DataArray`. For historical reasons DataArrays are often -referred to as "Datasets" in SatPy. These objects act similar to normal +referred to as "Datasets" in Satpy. These objects act similar to normal numpy arrays, but add additional metadata and attributes for describing the data. Metadata is stored in a ``.attrs`` dictionary and named dimensions can be accessed in a ``.dims`` attribute, along with other attributes. @@ -41,14 +41,14 @@ with special care taken to make sure the metadata dictionary contains expected values. See the XArray documentation for more info on handling :class:`xarray.DataArray` objects. -Additionally, SatPy uses a special form of DataArrays where data is stored -in :class:`dask.array.Array` objects which allows SatPy to perform +Additionally, Satpy uses a special form of DataArrays where data is stored +in :class:`dask.array.Array` objects which allows Satpy to perform multi-threaded lazy operations vastly improving the performance of processing. For help on developing with dask and xarray see :doc:`dev_guide/xarray_migration` or the documentation for the specific project. -To uniquely identify ``DataArray`` objects SatPy uses `DatasetID`. A +To uniquely identify ``DataArray`` objects Satpy uses `DatasetID`. A ``DatasetID`` consists of various pieces of available metadata. This usually includes `name` and `wavelength` as identifying metadata, but also includes `resolution`, `calibration`, `polarization`, and additional `modifiers` @@ -57,18 +57,18 @@ to further distinguish one dataset from another. .. warning:: XArray includes other object types called "Datasets". These are different - from the "Datasets" mentioned in SatPy. + from the "Datasets" mentioned in Satpy. Reading ======= -One of the biggest advantages of using SatPy is the large number of input +One of the biggest advantages of using Satpy is the large number of input file formats that it can read. It encapsulates this functionality into -individual :doc:`readers`. SatPy Readers handle all of the complexity of +individual :doc:`readers`. Satpy Readers handle all of the complexity of reading whatever format they represent. Meteorological Satellite file formats can be extremely complex and formats are rarely reused across satellites -or instruments. No matter the format, SatPy's Reader interface is meant to +or instruments. No matter the format, Satpy's Reader interface is meant to provide a consistent data loading interface while still providing flexibility to add new complex file formats. @@ -78,10 +78,10 @@ Compositing Many users of satellite imagery combine multiple sensor channels to bring out certain features of the data. This includes using one dataset to enhance another, combining 3 or more datasets in to an RGB image, or any other -combination of datasets. SatPy comes with a lot of common composite +combination of datasets. Satpy comes with a lot of common composite combinations built-in and allows the user to request them like any other -dataset. SatPy also makes it possible to create your own custom composites -and have SatPy treat them like any other dataset. See :doc:`composites` +dataset. Satpy also makes it possible to create your own custom composites +and have Satpy treat them like any other dataset. See :doc:`composites` for more information. Resampling @@ -93,9 +93,9 @@ coordinates. It is also common to see the channels from a single sensor in multiple resolutions, making it complicated to combine or compare the datasets. Many use cases of satellite data require the data to be in a certain projection other than the native projection or to have -output imagery cover a specific area of interest. SatPy makes it easy to +output imagery cover a specific area of interest. Satpy makes it easy to resample datasets to allow for users to combine them or grid them to these -projections or areas of interest. SatPy uses the PyTroll `pyresample` package +projections or areas of interest. Satpy uses the PyTroll `pyresample` package to provide nearest neighbor, bilinear, or elliptical weighted averaging resampling methods. See :doc:`resample` for more information. @@ -104,13 +104,13 @@ Enhancements When making images from satellite data the data has to be manipulated to be compatible with the output image format and still look good to the human eye. -SatPy calls this functionality "enhancing" the data, also commonly called +Satpy calls this functionality "enhancing" the data, also commonly called scaling or stretching the data. This process can become complicated not just because of how subjective the quality of an image can be, but also because of historical expectations of forecasters and other users for how the data -should look. SatPy tries to hide the complexity of all the possible +should look. Satpy tries to hide the complexity of all the possible enhancement methods from the user and just provide the best looking image -by default. SatPy still makes it possible to customize these procedures, but +by default. Satpy still makes it possible to customize these procedures, but in most cases it shouldn't be necessary. See the documentation on :doc:`writers` for more information on what's possible for output formats and enhancing images. @@ -118,9 +118,9 @@ and enhancing images. Writing ======= -SatPy is designed to make data loading, manipulating, and analysis easy. +Satpy is designed to make data loading, manipulating, and analysis easy. However, the best way to get satellite imagery data out to as many users -as possible is to make it easy to save it in multiple formats. SatPy allows +as possible is to make it easy to save it in multiple formats. Satpy allows users to save data in image formats like PNG or GeoTIFF as well as data file formats like NetCDF. Each format's complexity is hidden behind the interface of individual Writer objects and includes keyword arguments for accessing diff --git a/doc/source/quickstart.rst b/doc/source/quickstart.rst index 4ef4411004..ef8c8c8a09 100644 --- a/doc/source/quickstart.rst +++ b/doc/source/quickstart.rst @@ -11,11 +11,11 @@ Loading and accessing data >>> sys.setdefaultencoding('utf8') To work with weather satellite data you must create a -:class:`~satpy.scene.Scene` object. SatPy does not currently provide an +:class:`~satpy.scene.Scene` object. Satpy does not currently provide an interface to download satellite data, it assumes that the data is on a -local hard disk already. In order for SatPy to get access to the data the +local hard disk already. In order for Satpy to get access to the data the Scene must be told what files to read and what -:ref:`SatPy Reader ` should read them: +:ref:`Satpy Reader ` should read them: >>> from satpy import Scene >>> from glob import glob @@ -101,7 +101,7 @@ method. Printing the Scene object will list each of the modifiers: () ancillary_variables: [] -SatPy allows loading file data by wavelengths in micrometers (shown above) or by channel name:: +Satpy allows loading file data by wavelengths in micrometers (shown above) or by channel name:: >>> global_scene.load(["VIS006", "VIS008", "IR_108"]) @@ -163,7 +163,7 @@ advanced loading methods see the :doc:`readers` documentation. Generating composites ===================== -SatPy comes with many composite recipes built-in and makes them loadable like any other dataset: +Satpy comes with many composite recipes built-in and makes them loadable like any other dataset: >>> global_scene.load(['overview']) @@ -204,7 +204,7 @@ to a uniform grid, limiting input data to an area of interest, changing from one projection to another, or for preparing datasets to be combined in a composite (see above). For more details on resampling, different resampling algorithms, and creating your own area of interest see the -:doc:`resample` documentation. To resample a SatPy Scene: +:doc:`resample` documentation. To resample a Satpy Scene: >>> local_scene = global_scene.resample("eurol") @@ -250,10 +250,10 @@ function called `check_satpy` can be used: >>> from satpy.config import check_satpy >>> check_satpy() -Due to the way SatPy works, producing as many datasets as possible, there are +Due to the way Satpy works, producing as many datasets as possible, there are times that behavior can be unexpected but with no exceptions raised. To help troubleshoot these situations log messages can be turned on. To do this run -the following code before running any other SatPy code: +the following code before running any other Satpy code: >>> from satpy.utils import debug_on >>> debug_on() diff --git a/doc/source/readers.rst b/doc/source/readers.rst index eae5d4824e..594eac1574 100644 --- a/doc/source/readers.rst +++ b/doc/source/readers.rst @@ -6,7 +6,7 @@ Readers How to read cloud products from NWCSAF software. (separate document?) -SatPy supports reading and loading data from many input file formats and +Satpy supports reading and loading data from many input file formats and schemes. The :class:`~satpy.scene.Scene` object provides a simple interface around all the complexity of these various formats through its ``load`` method. The following sections describe the different way data can be loaded, @@ -28,7 +28,7 @@ Coming soon... Load data ========= -Datasets in SatPy are identified by certain pieces of metadata set during +Datasets in Satpy are identified by certain pieces of metadata set during data loading. These include `name`, `wavelength`, `calibration`, `resolution`, `polarization`, and `modifiers`. Normally, once a ``Scene`` is created requesting datasets by `name` or `wavelength` is all that is @@ -43,7 +43,7 @@ However, in many cases datasets are available in multiple spatial resolutions, multiple calibrations (``brightness_temperature``, ``reflectance``, ``radiance``, etc), multiple polarizations, or have corrections or other modifiers already applied -to them. By default SatPy will provide the version of the dataset with the +to them. By default Satpy will provide the version of the dataset with the highest resolution and the highest level of calibration (brightness temperature or reflectance over radiance). It is also possible to request one of these exact versions of a dataset by using the @@ -63,7 +63,7 @@ Or multiple calibrations:: >>> scn.load([0.6, 10.8], calibrations=['brightness_temperature', 'radiance']) -In the above case SatPy will load whatever dataset is available and matches +In the above case Satpy will load whatever dataset is available and matches the specified parameters. So the above ``load`` call would load the ``0.6`` (a visible/reflectance band) radiance data and ``10.8`` (an IR band) brightness temperature data. @@ -89,7 +89,7 @@ names of Datasets:: Search for local files ====================== -SatPy provides a utility +Satpy provides a utility :func:`~satpy.readers.find_files_and_readers` for searching for files in a base directory matching various search parameters. This function discovers files based on filename patterns. It returns a dictionary mapping reader name @@ -109,7 +109,7 @@ the :class:`~satpy.scene.Scene` initialization. See the :func:`~satpy.readers.find_files_and_readers` documentation for more information on the possible parameters. -Adding a Reader to SatPy +Adding a Reader to Satpy ======================== This is described in the developer guide, see :doc:`dev_guide/custom_reader`. diff --git a/doc/source/writers.rst b/doc/source/writers.rst index 9bfbb68103..1b1a786d50 100644 --- a/doc/source/writers.rst +++ b/doc/source/writers.rst @@ -2,7 +2,7 @@ Writers ======= -SatPy makes it possible to save datasets in multiple formats. For details +Satpy makes it possible to save datasets in multiple formats. For details on additional arguments and features available for a specific Writer see the table below. Most use cases will want to save datasets using the :meth:`~satpy.scene.Scene.save_datasets` method:: @@ -24,7 +24,7 @@ One common parameter across almost all Writers is ``filename`` and .. _writer_table: -.. list-table:: SatPy Writers +.. list-table:: Satpy Writers :header-rows: 1 * - Description diff --git a/satpy/__init__.py b/satpy/__init__.py index 0c5e2c49c9..9582acc067 100644 --- a/satpy/__init__.py +++ b/satpy/__init__.py @@ -26,7 +26,7 @@ # You should have received a copy of the GNU General Public License # along with satpy. If not, see . -"""SatPy Package initializer. +"""Satpy Package initializer. """ import os diff --git a/satpy/composites/__init__.py b/satpy/composites/__init__.py index 4d67eeedf1..032177307c 100644 --- a/satpy/composites/__init__.py +++ b/satpy/composites/__init__.py @@ -1,7 +1,7 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- -# Copyright (c) 2015-2018 PyTroll developers +# Copyright (c) 2015-2019 PyTroll developers # Author(s): @@ -902,7 +902,7 @@ def __call__(self, projectables, **info): class DayNightCompositor(GenericCompositor): """A compositor that blends a day data with night data.""" - def __init__(self, lim_low=85., lim_high=95., **kwargs): + def __init__(self, name, lim_low=85., lim_high=95., **kwargs): """Collect custom configuration values. Args: @@ -913,7 +913,7 @@ def __init__(self, lim_low=85., lim_high=95., **kwargs): """ self.lim_low = lim_low self.lim_high = lim_high - super(DayNightCompositor, self).__init__(**kwargs) + super(DayNightCompositor, self).__init__(name, **kwargs) def __call__(self, projectables, **kwargs): projectables = self.check_areas(projectables) @@ -1132,7 +1132,7 @@ def __call__(self, projectables, *args, **kwargs): class CloudCompositor(GenericCompositor): - def __init__(self, transition_min=258.15, transition_max=298.15, + def __init__(self, name, transition_min=258.15, transition_max=298.15, transition_gamma=3.0, **kwargs): """Collect custom configuration values. @@ -1147,7 +1147,7 @@ def __init__(self, transition_min=258.15, transition_max=298.15, self.transition_min = transition_min self.transition_max = transition_max self.transition_gamma = transition_gamma - super(CloudCompositor, self).__init__(**kwargs) + super(CloudCompositor, self).__init__(name, **kwargs) def __call__(self, projectables, **kwargs): @@ -1314,7 +1314,6 @@ class SelfSharpenedRGB(RatioSharpenedRGB): new_G = G * ratio new_B = B * ratio - """ @staticmethod diff --git a/satpy/composites/cloud_products.py b/satpy/composites/cloud_products.py index cd0e3e4607..96d7dd6347 100644 --- a/satpy/composites/cloud_products.py +++ b/satpy/composites/cloud_products.py @@ -1,7 +1,7 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- -# Copyright (c) 2015-2018 PyTroll developers +# Copyright (c) 2015-2019 PyTroll developers # Author(s): @@ -28,6 +28,7 @@ import xarray as xr from satpy.composites import ColormapCompositor +from satpy.composites import GenericCompositor class CloudTopHeightCompositor(ColormapCompositor): @@ -76,3 +77,46 @@ def __call__(self, projectables, **info): res = super(CloudTopHeightCompositor, self).__call__(chans, **data.attrs) res.attrs['_FillValue'] = np.nan return res + + +class PrecipCloudsRGB(GenericCompositor): + + def __call__(self, projectables, *args, **kwargs): + """Make an RGB image out of the three probability categories of the NWCSAF precip product.""" + + projectables = self.check_areas(projectables) + light = projectables[0] + moderate = projectables[1] + intense = projectables[2] + status_flag = projectables[3] + + if np.bitwise_and(status_flag, 4).any(): + # AMSU is used + maxs1 = 70 + maxs2 = 70 + maxs3 = 100 + else: + # avhrr only + maxs1 = 30 + maxs2 = 50 + maxs3 = 40 + + scalef3 = 1.0 / maxs3 - 1 / 255.0 + scalef2 = 1.0 / maxs2 - 1 / 255.0 + scalef1 = 1.0 / maxs1 - 1 / 255.0 + + p1data = (light*scalef1).where(light != 0) + p1data = p1data.where(light != light.attrs['_FillValue']) + p1data.attrs = light.attrs + data = moderate*scalef2 + p2data = data.where(moderate != 0) + p2data = p2data.where(moderate != moderate.attrs['_FillValue']) + p2data.attrs = moderate.attrs + data = intense*scalef3 + p3data = data.where(intense != 0) + p3data = p3data.where(intense != intense.attrs['_FillValue']) + p3data.attrs = intense.attrs + + res = super(PrecipCloudsRGB, self).__call__((p3data, p2data, p1data), + *args, **kwargs) + return res diff --git a/satpy/config.py b/satpy/config.py index a57279e74a..fb30d2dcc1 100644 --- a/satpy/config.py +++ b/satpy/config.py @@ -23,7 +23,7 @@ # You should have received a copy of the GNU General Public License # along with satpy. If not, see . -"""SatPy Configuration directory and file handling +"""Satpy Configuration directory and file handling """ import glob import logging diff --git a/satpy/dataset.py b/satpy/dataset.py index 677b447458..6a4318d571 100644 --- a/satpy/dataset.py +++ b/satpy/dataset.py @@ -145,7 +145,7 @@ class DatasetID(DatasetID): and "reflectance". If an element is `None` then it is considered not applicable. - A DatasetID can also be used in SatPy to query for a Dataset. This way + A DatasetID can also be used in Satpy to query for a Dataset. This way a fully qualified DatasetID can be found even if some of the DatasetID elements are unknown. In this case a `None` signifies something that is unknown or not applicable to the requested Dataset. diff --git a/satpy/demo/__init__.py b/satpy/demo/__init__.py new file mode 100644 index 0000000000..64f01e612f --- /dev/null +++ b/satpy/demo/__init__.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# +# Copyright (c) 2019 Satpy developers +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +"""Demo data download helper functions. + +Each ``get_*`` function below downloads files to a local directory and returns +a list of paths to those files. Some (not all) functions have multiple options +for how the data is downloaded (via the ``method`` keyword argument) +including: + +- gcsfs: Download data from a public google cloud storage bucket using the + ``gcsfs`` package. +- unidata_thredds: Access data using OpenDAP or similar method from Unidata's + public THREDDS server + (https://thredds.unidata.ucar.edu/thredds/catalog.html). +- uwaos_thredds: Access data using OpenDAP or similar method from the + University of Wisconsin - Madison's AOS department's THREDDS server. +- http: A last resort download method when nothing else is available of a + tarball or zip file from one or more servers available to the Satpy + project. +- uw_arcdata: A network mount available on many servers at the Space Science + and Engineering Center (SSEC) at the University of Wisconsin - Madison. + This is method is mainly meant when tutorials are taught at the SSEC + using a Jupyter Hub server. + +To use these functions, do: + + >>> from satpy import Scene, demo + >>> filenames = demo.get_us_midlatitude_cyclone_abi() + >>> scn = Scene(reader='abi_l1b', filenames=filenames) + +""" + +import os +import logging + +LOG = logging.getLogger(__name__) + + +def makedirs(directory, exist_ok=False): + """Python 2.7 friendly os.makedirs. + + After python 2.7 is dropped, just use `os.makedirs` with `existsok=True`. + """ + try: + os.makedirs(directory) + except OSError: + if not exist_ok: + raise + + +def get_us_midlatitude_cyclone_abi(base_dir='.', method=None, force=False): + """Get GOES-16 ABI (CONUS sector) data from March 14th 00:00Z. + + Args: + base_dir (str): Base directory for downloaded files. + method (str): Force download method for the data if not already cached. + Allowed options are: 'gcsfs'. Default of ``None`` will + choose the best method based on environment settings. + force (bool): Force re-download of data regardless of its existence on + the local system. Warning: May delete non-demo files stored in + download directory. + + """ + if method is None: + method = 'gcsfs' + if method not in ['gcsfs']: + raise NotImplementedError("Demo data download method '{}' not " + "implemented yet.".format(method)) + + from .google_cloud_platform import get_bucket_files + patterns = ['gs://gcp-public-data-goes-16/ABI-L1b-RadC/2019/073/00/*0002*.nc'] + subdir = os.path.join(base_dir, 'abi_l1b', '20190314_us_midlatitude_cyclone') + makedirs(subdir, exist_ok=True) + filenames = get_bucket_files(patterns, subdir, force=force) + assert len(filenames) == 16, "Not all ABI files could be downloaded" + return filenames diff --git a/satpy/demo/google_cloud_platform.py b/satpy/demo/google_cloud_platform.py new file mode 100644 index 0000000000..40c8f78f88 --- /dev/null +++ b/satpy/demo/google_cloud_platform.py @@ -0,0 +1,84 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# +# Copyright (c) 2019 Satpy developers +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +import os +import logging + +try: + from urllib.request import urlopen + from urllib.error import URLError +except ImportError: + # python 2 + from urllib2 import urlopen, URLError + +try: + import gcsfs +except ImportError: + gcsfs = None + +LOG = logging.getLogger(__name__) + + +def is_google_cloud_instance(): + try: + return urlopen('http://metadata.google.internal').headers.get('Metadata-Flavor') == 'Google' + except URLError: + return False + + +def get_bucket_files(glob_pattern, base_dir, force=False): + """Helper function to download files from Google Cloud Storage. + + Args: + glob_pattern (str or list): Glob pattern string or series of patterns + used to search for on Google Cloud Storage. The pattern should + include the "gs://" protocol prefix. + base_dir (str): Root directory to place downloaded files on the local + system. + force (bool): Force re-download of data regardless of its existence on + the local system. Warning: May delete non-demo files stored in + download directory. + + """ + if gcsfs is None: + raise RuntimeError("Missing 'gcsfs' dependency for GCS download.") + if not os.path.isdir(base_dir): + # it is the caller's responsibility to make this + raise OSError("Directory does not exist: {}".format(base_dir)) + + if isinstance(glob_pattern, str): + glob_pattern = [glob_pattern] + + fs = gcsfs.GCSFileSystem(token='anon') + filenames = [] + for gp in glob_pattern: + for fn in fs.glob(gp): + ondisk_fn = os.path.basename(fn) + ondisk_pathname = os.path.join(base_dir, ondisk_fn) + filenames.append(ondisk_pathname) + LOG.info("Downloading: {}".format(ondisk_pathname)) + + if force and os.path.isfile(ondisk_pathname): + os.remove(ondisk_pathname) + elif os.path.isfile(ondisk_pathname): + continue + fs.get('gs://' + fn, ondisk_pathname) + + if not filenames: + raise OSError("No files could be found or downloaded.") + return filenames diff --git a/satpy/etc/composites/modis.yaml b/satpy/etc/composites/modis.yaml index a7bfa6762c..8dc97c3569 100644 --- a/satpy/etc/composites/modis.yaml +++ b/satpy/etc/composites/modis.yaml @@ -90,3 +90,17 @@ composites: - name: '3' modifiers: [sunz_corrected, rayleigh_corrected] standard_name: ocean_color + + night_fog: + compositor: !!python/name:satpy.composites.GenericCompositor + prerequisites: + - compositor: !!python/name:satpy.composites.DifferenceCompositor + prerequisites: + - 12.0 + - 10.8 + - compositor: !!python/name:satpy.composites.DifferenceCompositor + prerequisites: + - 10.8 + - 3.75 + - 10.8 + standard_name: night_fog diff --git a/satpy/etc/composites/virr.yaml b/satpy/etc/composites/virr.yaml new file mode 100644 index 0000000000..be42c6110e --- /dev/null +++ b/satpy/etc/composites/virr.yaml @@ -0,0 +1,43 @@ +sensor_name: visir/virr + +modifiers: + sunz_corrected: + compositor: !!python/name:satpy.composites.SunZenithCorrector + prerequisites: + - name: solar_zenith_angle + + rayleigh_corrected: + compositor: !!python/name:satpy.composites.PSPRayleighReflectance + atmosphere: us-standard + aerosol_type: rayleigh_only + prerequisites: + - name: R1 + modifiers: [sunz_corrected] + optional_prerequisites: + - name: satellite_azimuth_angle + - name: satellite_zenith_angle + - name: solar_azimuth_angle + - name: solar_zenith_angle + +composites: + true_color_raw: + compositor: !!python/name:satpy.composites.GenericCompositor + prerequisites: + - name: R1 + modifiers: [sunz_corrected] + - name: R6 + modifiers: [sunz_corrected] + - name: R4 + modifiers: [sunz_corrected] + standard_name: true_color + + true_color: + compositor: !!python/name:satpy.composites.GenericCompositor + prerequisites: + - name: R1 + modifiers: [sunz_corrected, rayleigh_corrected] + - name: R6 + modifiers: [sunz_corrected, rayleigh_corrected] + - name: R4 + modifiers: [sunz_corrected, rayleigh_corrected] + standard_name: true_color diff --git a/satpy/etc/composites/visir.yaml b/satpy/etc/composites/visir.yaml index 1bf58388a1..4982b312ba 100644 --- a/satpy/etc/composites/visir.yaml +++ b/satpy/etc/composites/visir.yaml @@ -269,6 +269,15 @@ composites: - night_fog - solar_zenith_angle + precipitation_probability: + compositor: !!python/name:satpy.composites.cloud_products.PrecipCloudsRGB + prerequisites: + - pc_precip_light + - pc_precip_moderate + - pc_precip_intense + - pc_status_flag + standard_name: precipitation_probability + cloudmask: compositor: !!python/name:satpy.composites.PaletteCompositor prerequisites: diff --git a/satpy/etc/readers/generic_image.yaml b/satpy/etc/readers/generic_image.yaml index 11b84a04e4..5da1bb3ff1 100644 --- a/satpy/etc/readers/generic_image.yaml +++ b/satpy/etc/readers/generic_image.yaml @@ -15,11 +15,32 @@ file_types: file_reader: !!python/name:satpy.readers.generic_image.GenericImageFileHandler file_patterns: - '{start_time:%Y%m%d_%H%M}{filename}.png' + - '{start_time:%Y%m%d_%H%M}{filename}.PNG' - '{start_time:%Y%m%d_%H%M}{filename}.jpg' + - '{start_time:%Y%m%d_%H%M}{filename}.jpeg' + - '{start_time:%Y%m%d_%H%M}{filename}.JPG' + - '{start_time:%Y%m%d_%H%M}{filename}.JPEG' - '{start_time:%Y%m%d_%H%M}{filename}.tif' + - '{start_time:%Y%m%d_%H%M}{filename}.tiff' + - '{start_time:%Y%m%d_%H%M}{filename}.TIF' + - '{start_time:%Y%m%d_%H%M}{filename}.TIFF' - '{filename}{start_time:%Y%m%d_%H%M}.png' + - '{filename}{start_time:%Y%m%d_%H%M}.PNG' - '{filename}{start_time:%Y%m%d_%H%M}.jpg' + - '{filename}{start_time:%Y%m%d_%H%M}.jpeg' + - '{filename}{start_time:%Y%m%d_%H%M}.JPG' + - '{filename}{start_time:%Y%m%d_%H%M}.JPEG' - '{filename}{start_time:%Y%m%d_%H%M}.tif' + - '{filename}{start_time:%Y%m%d_%H%M}.tiff' + - '{filename}{start_time:%Y%m%d_%H%M}.TIF' + - '{filename}{start_time:%Y%m%d_%H%M}.TIFF' - '{filename}.png' + - '{filename}.PNG' - '{filename}.jpg' + - '{filename}.jpeg' + - '{filename}.JPG' + - '{filename}.JPEG' - '{filename}.tif' + - '{filename}.tiff' + - '{filename}.TIF' + - '{filename}.TIFF' diff --git a/satpy/etc/readers/safe_sar_l2_ocn.yaml b/satpy/etc/readers/safe_sar_l2_ocn.yaml new file mode 100644 index 0000000000..24db9a2586 --- /dev/null +++ b/satpy/etc/readers/safe_sar_l2_ocn.yaml @@ -0,0 +1,169 @@ +reader: + description: SAFE Reader for SAR L2 OCN data + name: safe_sar_l2_ocn + sensors: [sar-c] + default_channels: [] + reader: !!python/name:satpy.readers.yaml_reader.FileYAMLReader + +file_types: + safe_measurement: + file_reader: !!python/name:satpy.readers.safe_sar_l2_ocn.SAFENC + file_patterns: ['{fmission_id:3s}_{fsar_mode:2s}_{fproduct_type:3s}{fresolution:1s}_{fprocessing_level:1s}{fproduct_class:1s}{fpolarization:2s}_{fstart_time:%Y%m%dT%H%M%S}_{fend_time:%Y%m%dT%H%M%S}_{forbit_number:6d}_{fmission_data_take_id:6s}_{fproduct_unique_id:4s}.SAFE/measurement/{mission_id:3s}-{swath_id:2s}-{product_type:3s}-{polarization:2s}-{start_time:%Y%m%dt%H%M%S}-{end_time:%Y%m%dt%H%M%S}-{orbit_number:6d}-{mission_data_take_id:6s}-{image_number:3s}.nc'] + +datasets: + owiLat: + name: owiLat + file_type: safe_measurement + standard_name: latitude + units: degree + + owiLon: + name: owiLon + file_type: safe_measurement + standard_name: longitude + units: degree + + owiWindDirection: + name: owiWindDirection + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: degree + + owiWindSpeed: + name: owiWindSpeed + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m s-1 + + owiEcmwfWindDirection: + name: owiEcmwfWindDirection + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: degree + + owiEcmwfWindSpeed: + name: owiEcmwfWindSpeed + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m s-1 + + owiHs: + name: owiHs + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m + + owiWl: + name: owiWl + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m + + owiDirmet: + name: owiDirmet + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: degrees + + owiWindSeaHs: + name: owiWindSeaHs + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m + + owiIncidenceAngle: + name: owiIncidenceAngle + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: degrees + + owiElevationAngle: + name: owiElevationAngle + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: degrees + + owiNrcs: + name: owiNrcs + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m2/m2 + + owiNesz: + name: owiNesz + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m2/m2 + + owiNrcsNeszCorr: + name: owiNrcsNeszCorr + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m2/m2 + + owiPolarisationName: + name: owiPolarisationName + sensor: sar-c + file_type: safe_measurement + + owiPBright: + name: owiPBright + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: '%' + + owiNrcsCmod: + name: owiNrcsCmod + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: m2/m2 + + owiCalConstObsi: + name: owiCalConstObsi + sensor: sar-c + file_type: safe_measurement + + owiCalConstInci: + name: owiCalConstInci + sensor: sar-c + file_type: safe_measurement + + owiInversionQuality: + name: owiInversionQuality + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + + owiMask: + name: owiMask + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + + owiHeading: + name: owiHeading + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] + units: degrees + + owiWindQuality: + name: owiWindQuality + sensor: sar-c + file_type: safe_measurement + coordinates: [owiLon, owiLat] diff --git a/satpy/etc/readers/viirs_edr_active_fires.yaml b/satpy/etc/readers/viirs_edr_active_fires.yaml new file mode 100644 index 0000000000..24b0871c84 --- /dev/null +++ b/satpy/etc/readers/viirs_edr_active_fires.yaml @@ -0,0 +1,48 @@ +reader: + description: VIIRS Active Fires Reader + name: viirs_edr_active_fires + reader: !!python/name:satpy.readers.yaml_reader.FileYAMLReader + sensors: [viirs] + +file_types: + fires_netcdf: + file_reader: !!python/name:satpy.readers.viirs_edr_active_fires.VIIRSActiveFiresFileHandler + file_patterns: + - 'AFEDR_npp_d{start_time:%Y%m%d_t%H%M%S%f}_e{end_time:%H%M%S%f}_b{orbit:5d}_c{creation_time}_{source}.nc' + fires_text: + file_reader: !!python/name:satpy.readers.viirs_edr_active_fires.VIIRSActiveFiresTextFileHandler + file_patterns: + - 'AFEDR_npp_d{start_time:%Y%m%d_t%H%M%S%f}_e{end_time:%H%M%S%f}_b{orbit:5d}_c{creation_time}_{source}.txt' + +datasets: + detection_confidence: + name: detection_confidence + standard_name: fire_confidence + file_type: [fires_netcdf, fires_text] + file_key: "Fire Pixels/FP_confidence" + coordinates: [longitude, latitude] + units: '%' + longitude: + name: longitude + standard_name: longitude + file_type: [fires_netcdf, fires_text] + file_key: "Fire Pixels/FP_longitude" + units: 'degrees' + latitude: + name: latitude + standard_name: latitude + file_type: [fires_netcdf, fires_text] + file_key: "Fire Pixels/FP_latitude" + units: 'degrees' + power: + name: power + file_type: [fires_netcdf, fires_text] + file_key: "Fire Pixels/FP_power" + coordinates: [longitude, latitude] + units: 'MW' + T13: + name: T13 + file_type: [fires_netcdf, fires_text] + file_key: "Fire Pixels/FP_T13" + coordinates: [longitude, latitude] + units: 'K' \ No newline at end of file diff --git a/satpy/etc/readers/virr_l1b.yaml b/satpy/etc/readers/virr_l1b.yaml new file mode 100644 index 0000000000..890cf518b8 --- /dev/null +++ b/satpy/etc/readers/virr_l1b.yaml @@ -0,0 +1,180 @@ +reader: + description: reader for VIRR data + name: virr_l1b + sensors: [virr] + reader: !!python/name:satpy.readers.yaml_reader.FileYAMLReader + +file_types: + virr_l1b: + file_reader: !!python/name:satpy.readers.virr_l1b.VIRR_L1B + file_patterns: + - 'tf{creation_time:%Y%j%H%M%S}.{platform_id}-L_VIRRX_L1B.HDF' + geolocation_prefix: '' + virr_geoxx: + file_reader: !!python/name:satpy.readers.virr_l1b.VIRR_L1B + file_patterns: + - 'tf{creation_time:%Y%j%H%M%S}.{platform_id}-L_VIRRX_GEOXX.HDF' + geolocation_prefix: 'Geolocation/' + +datasets: + R1: + name: R1 + wavelength: [0.58, 0.63, 0.68] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_RefSB + band_index: 0 + standard_name: toa_bidirectional_reflectance + coordinates: [longitude, latitude] + calibration: reflectance + level: 1 + + R2: + name: R2 + wavelength: [0.84, 0.865, 0.89] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_RefSB + band_index: 1 + standard_name: toa_bidirectional_reflectance + coordinates: [longitude, latitude] + calibration: reflectance + level: 1 + + E1: + name: E1 + wavelength: [3.55, 3.74, 3.93] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_Emissive + band_index: 0 + standard_name: toa_brightness_temperature + coordinates: [longitude, latitude] + calibration: brightness_temperature + level: 1 + + E2: + name: E2 + wavelength: [10.3, 10.8, 11.3] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_Emissive + band_index: 1 + standard_name: toa_brightness_temperature + coordinates: [longitude, latitude] + calibration: brightness_temperature + level: 1 + + E3: + name: E3 + wavelength: [11.5, 12.0, 12.5] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_Emissive + band_index: 2 + standard_name: toa_brightness_temperature + coordinates: [longitude, latitude] + calibration: brightness_temperature + level: 1 + + R3: + name: R3 + wavelength: [1.55, 1.6, 1.64] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_RefSB + band_index: 2 + standard_name: toa_bidirectional_reflectance + coordinates: [longitude, latitude] + calibration: reflectance + level: 1 + + R4: + name: R4 + wavelength: [0.43, 0.455, 0.48] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_RefSB + band_index: 3 + standard_name: toa_bidirectional_reflectance + coordinates: [longitude, latitude] + calibration: reflectance + level: 1 + + R5: + name: R5 + wavelength: [0.48, 0.505, 0.53] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_RefSB + band_index: 4 + standard_name: toa_bidirectional_reflectance + coordinates: [longitude, latitude] + calibration: reflectance + level: 1 + + R6: + name: R6 + wavelength: [0.53, 0.555, 0.58] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_RefSB + band_index: 5 + standard_name: toa_bidirectional_reflectance + coordinates: [longitude, latitude] + calibration: reflectance + level: 1 + + R7: + name: R7 + wavelength: [1.325, 1.36, 1.395] + resolution: 1000 + file_type: virr_l1b + file_key: Data/EV_RefSB + band_index: 6 + standard_name: toa_bidirectional_reflectance + coordinates: [longitude, latitude] + calibration: reflectance + level: 1 + + satellite_azimuth_angle: + name: satellite_azimuth_angle + file_type: [virr_geoxx, virr_l1b] + file_key: SensorAzimuth + standard_name: sensor_azimuth_angle + coordinates: [longitude, latitude] + + satellite_zenith_angle: + name: satellite_zenith_angle + file_type: [virr_geoxx, virr_l1b] + file_key: SensorZenith + standard_name: sensor_zenith_angle + coordinates: [longitude, latitude] + + solar_azimuth_angle: + name: solar_azimuth_angle + file_type: [virr_geoxx, virr_l1b] + file_key: SolarAzimuth + standard_name: solar_azimuth_angle + coordinates: [longitude, latitude] + + solar_zenith_angle: + name: solar_zenith_angle + file_type: [virr_geoxx, virr_l1b] + file_key: SolarZenith + standard_name: solar_zenith_angle + coordinates: [longitude, latitude] + + longitude: + name: longitude + resolution: 1000 + file_type: [virr_l1b, virr_geoxx] + file_key: Longitude + standard_name: longitude + + latitude: + name: latitude + resolution: 1000 + file_type: [virr_l1b, virr_geoxx] + file_key: Latitude + standard_name: latitude \ No newline at end of file diff --git a/satpy/etc/writers/scmi.yaml b/satpy/etc/writers/scmi.yaml index 55f21c4546..a54cffc95b 100644 --- a/satpy/etc/writers/scmi.yaml +++ b/satpy/etc/writers/scmi.yaml @@ -1,6 +1,6 @@ # Originally converted from the CSPP Polar2Grid SCMI Writer # Some datasets are named differently and have not been converted to -# SatPy-style naming yet. These config entries are commented out. +# Satpy-style naming yet. These config entries are commented out. writer: name: scmi description: AWIPS-compatible Tiled NetCDF4 Writer diff --git a/satpy/multiscene.py b/satpy/multiscene.py index 1b94fb770c..f4e738b5ba 100644 --- a/satpy/multiscene.py +++ b/satpy/multiscene.py @@ -45,6 +45,11 @@ except ImportError: imageio = None +try: + from dask.distributed import get_client +except ImportError: + get_client = None + log = logging.getLogger(__name__) @@ -389,17 +394,18 @@ def _get_animation_frames(self, all_datasets, shape, fill_value=None, def _get_client(self, client=True): """Determine what dask distributed client to use.""" client = client or None # convert False/None to None - if client is True: + if client is True and get_client is None: + log.debug("'dask.distributed' library was not found, will " + "use simple serial processing.") + client = None + elif client is True: try: # get existing client - from dask.distributed import get_client client = get_client() - except ImportError: - log.debug("'dask.distributed' library was not found, will " - "use simple serial processing.") except ValueError: log.warning("No dask distributed client was provided or found, " "but distributed features were requested. Will use simple serial processing.") + client = None return client def _distribute_frame_compute(self, writers, frame_keys, frames_to_write, client, batch_size=1): diff --git a/satpy/readers/__init__.py b/satpy/readers/__init__.py index 793e15fb05..2b9dfda92b 100644 --- a/satpy/readers/__init__.py +++ b/satpy/readers/__init__.py @@ -440,9 +440,9 @@ def group_files(files_to_sort, reader=None, time_threshold=10, that files will not be grouped properly (datetimes being barely unequal). Defaults to a reader's ``group_keys`` configuration (set in YAML), otherwise ``('start_time',)``. - ppp_config_dir (str): Root usser configuration directory for SatPy. + ppp_config_dir (str): Root usser configuration directory for Satpy. This will be deprecated in the future, but is here for consistency - with other SatPy features. + with other Satpy features. reader_kwargs (dict): Additional keyword arguments to pass to reader creation. @@ -559,10 +559,10 @@ def configs_for_reader(reader=None, ppp_config_dir=None): continue new_name = OLD_READER_NAMES[reader_name] - # SatPy 0.11 only displays a warning - # SatPy 0.13 will raise an exception + # Satpy 0.11 only displays a warning + # Satpy 0.13 will raise an exception raise ValueError("Reader name '{}' has been deprecated, use '{}' instead.".format(reader_name, new_name)) - # SatPy 0.15 or 1.0, remove exception and mapping + # Satpy 0.15 or 1.0, remove exception and mapping reader = new_readers # given a config filename or reader name @@ -638,7 +638,7 @@ def find_files_and_readers(start_time=None, end_time=None, base_dir=None, reader (str or list): The name of the reader to use for loading the data or a list of names. sensor (str or list): Limit used files by provided sensors. ppp_config_dir (str): The directory containing the configuration - files for SatPy. + files for Satpy. filter_parameters (dict): Filename pattern metadata to filter on. `start_time` and `end_time` are automatically added to this dictionary. Shortcut for `reader_kwargs['filter_parameters']`. @@ -699,8 +699,6 @@ def load_readers(filenames=None, reader=None, reader_kwargs=None, filenames (iterable or dict): A sequence of files that will be used to load data from. A ``dict`` object should map reader names to a list of filenames for that reader. reader (str or list): The name of the reader to use for loading the data or a list of names. - filter_parameters (dict): Specify loaded file filtering parameters. - Shortcut for `reader_kwargs['filter_parameters']`. reader_kwargs (dict): Keyword arguments to pass to specific reader instances. ppp_config_dir (str): The directory containing the configuration files for satpy. @@ -709,6 +707,9 @@ def load_readers(filenames=None, reader=None, reader_kwargs=None, """ reader_instances = {} reader_kwargs = reader_kwargs or {} + reader_kwargs_without_filter = reader_kwargs.copy() + reader_kwargs_without_filter.pop('filter_parameters', None) + if ppp_config_dir is None: ppp_config_dir = get_environ_config_dir() @@ -749,7 +750,7 @@ def load_readers(filenames=None, reader=None, reader_kwargs=None, if readers_files: loadables = reader_instance.select_files_from_pathnames(readers_files) if loadables: - reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs) + reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs_without_filter) reader_instances[reader_instance.name] = reader_instance remaining_filenames -= set(loadables) if not remaining_filenames: diff --git a/satpy/readers/abi_l1b.py b/satpy/readers/abi_l1b.py index 19410bffe2..b0f967916a 100644 --- a/satpy/readers/abi_l1b.py +++ b/satpy/readers/abi_l1b.py @@ -1,7 +1,7 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- # -# Copyright (c) 2016-2018 SatPy developers +# Copyright (c) 2016-2018 Satpy developers # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/satpy/readers/ahi_hsd.py b/satpy/readers/ahi_hsd.py index a316d26c9e..8f466a5401 100644 --- a/satpy/readers/ahi_hsd.py +++ b/satpy/readers/ahi_hsd.py @@ -1,13 +1,7 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- # -# Copyright (c) 2014-2018 PyTroll developers -# -# Author(s): -# -# Adam.Dybbroe -# Cooke, Michael.C, UK Met Office -# Martin Raspaud +# Copyright (c) 2014-2019 PyTroll developers # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -45,6 +39,7 @@ import numpy as np import dask.array as da import xarray as xr +import warnings from pyresample import geometry from satpy import CHUNK_SIZE @@ -157,7 +152,10 @@ # Visible, near-infrared band (Band No. 1 – 6) # (Band No. 1: backup operation (See Table 4 bb)) _VISCAL_INFO_TYPE = np.dtype([("coeff_rad2albedo_conversion", "f8"), - ("spare", "S104"), + ("coeff_update_time", "f8"), + ("cali_gain_count2rad_conversion", "f8"), + ("cali_offset_count2rad_conversion", "f8"), + ("spare", "S80"), ]) # 6 Inter-calibration information block @@ -165,19 +163,19 @@ ("hblock_number", "u1"), ("blocklength", " 1: - return xr.DataArray(dset, dims=['y', 'x']) - else: - return xr.DataArray(dset) + dset_data = da.from_array(dset, chunks=CHUNK_SIZE) + if dset.ndim == 2: + return xr.DataArray(dset_data, dims=['y', 'x'], attrs=dset.attrs) + return xr.DataArray(dset_data) return val diff --git a/satpy/readers/hrit_base.py b/satpy/readers/hrit_base.py index c276dd3f49..2fd24ddc64 100644 --- a/satpy/readers/hrit_base.py +++ b/satpy/readers/hrit_base.py @@ -229,6 +229,7 @@ def _get_hd(self, hdr_info): 'h': 35785831.00, # FIXME: find a reasonable SSP 'SSP_longitude': 0.0} + self.mda['navigation_parameters'] = {} def get_shape(self, dsid, ds_info): return int(self.mda['number_of_lines']), int(self.mda['number_of_columns']) diff --git a/satpy/readers/msi_safe.py b/satpy/readers/msi_safe.py index f60e69582a..adee73b778 100644 --- a/satpy/readers/msi_safe.py +++ b/satpy/readers/msi_safe.py @@ -137,10 +137,10 @@ def get_area_def(self, dsid): self.tile, "On-the-fly area", self.tile, - proj_dict={'init': epsg}, - x_size=cols, - y_size=rows, - area_extent=area_extent) + {'init': epsg}, + cols, + rows, + area_extent) return area @staticmethod diff --git a/satpy/readers/safe_sar_l2_ocn.py b/satpy/readers/safe_sar_l2_ocn.py new file mode 100644 index 0000000000..a95bc38f3e --- /dev/null +++ b/satpy/readers/safe_sar_l2_ocn.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# Copyright (c) 2019 Pytroll developers + +# Author(s): + +# Trygve Aspenes + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +"""SAFE SAR L2 OCN format reader + +The OCN data contains various parameters, but mainly the wind speed and direction +calculated from SAR data and input model data from ECMWF + +Implemented in this reader is the OWI, Ocean Wind field. + +See more at ESA webpage https://sentinel.esa.int/web/sentinel/ocean-wind-field-component +""" + +import logging + +from satpy.readers.file_handlers import BaseFileHandler +from satpy import CHUNK_SIZE + +import numpy as np +import xarray as xr + +logger = logging.getLogger(__name__) + + +class SAFENC(BaseFileHandler): + """Measurement file reader.""" + + def __init__(self, filename, filename_info, filetype_info): + super(SAFENC, self).__init__(filename, filename_info, + filetype_info) + + self._start_time = filename_info['start_time'] + self._end_time = filename_info['end_time'] + # For some SAFE packages, fstart_time differs, but start_time is the same + # To avoid over writing exiting file with same start_time, a solution is to + # use fstart_time + self._fstart_time = filename_info['fstart_time'] + self._fend_time = filename_info['fend_time'] + + self._polarization = filename_info['polarization'] + + self.lats = None + self.lons = None + self._shape = None + self.area = None + + self.nc = xr.open_dataset(filename, + decode_cf=True, + mask_and_scale=False, + chunks={'owiAzSize': CHUNK_SIZE, + 'owiRaSize': CHUNK_SIZE}) + self.nc = self.nc.rename({'owiAzSize': 'y'}) + self.nc = self.nc.rename({'owiRaSize': 'x'}) + self.filename = filename + + def get_dataset(self, key, info): + """Load a dataset.""" + if key.name in ['owiLat', 'owiLon']: + if self.lons is None or self.lats is None: + self.lons = self.nc['owiLon'] + self.lats = self.nc['owiLat'] + if key.name == 'owiLat': + res = self.lats + else: + res = self.lons + res.attrs = info + else: + res = self.nc[key.name] + if key.name in ['owiHs', 'owiWl', 'owiDirmet']: + res = xr.DataArray(res, dims=['y', 'x', 'oswPartitions']) + elif key.name in ['owiNrcs', 'owiNesz', 'owiNrcsNeszCorr']: + res = xr.DataArray(res, dims=['y', 'x', 'oswPolarisation']) + elif key.name in ['owiPolarisationName']: + res = xr.DataArray(res, dims=['owiPolarisation']) + elif key.name in ['owiCalConstObsi', 'owiCalConstInci']: + res = xr.DataArray(res, dims=['owiIncSize']) + elif key.name.startswith('owi'): + res = xr.DataArray(res, dims=['y', 'x']) + else: + res = xr.DataArray(res, dims=['y', 'x']) + res.attrs.update(info) + if '_FillValue' in res.attrs: + res = res.where(res != res.attrs['_FillValue']) + res.attrs['_FillValue'] = np.nan + + if 'missionName' in self.nc.attrs: + res.attrs.update({'platform_name': self.nc.attrs['missionName']}) + + res.attrs.update({'fstart_time': self._fstart_time}) + res.attrs.update({'fend_time': self._fend_time}) + + if not self._shape: + self._shape = res.shape + + return res + + @property + def start_time(self): + """Product start_time, parsed from the measurement file name.""" + return self._start_time + + @property + def end_time(self): + """Product end_time, parsed from the measurement file name.""" + return self._end_time + + @property + def fstart_time(self): + """Product fstart_time meaning the start time parsed from the SAFE directory.""" + return self._fstart_time + + @property + def fend_time(self): + """Product fend_time meaning the end time parsed from the SAFE directory.""" + return self._fend_time diff --git a/satpy/readers/scmi.py b/satpy/readers/scmi.py index 52bb42d7ef..17ae2402f7 100644 --- a/satpy/readers/scmi.py +++ b/satpy/readers/scmi.py @@ -30,10 +30,10 @@ 1. Official SCMI format: NetCDF4 files where the main data variable is stored in a variable called "Sectorized_CMI". This variable name can be configured in the YAML configuration file. -2. SatPy/Polar2Grid SCMI format: NetCDF4 files based on the official SCMI +2. Satpy/Polar2Grid SCMI format: NetCDF4 files based on the official SCMI format created for the Polar2Grid project. This format was migrated to - SatPy as part of Polar2Grid's adoption of SatPy for the majority of its - features. This format is what is produced by SatPy's `scmi` writer. + Satpy as part of Polar2Grid's adoption of Satpy for the majority of its + features. This format is what is produced by Satpy's `scmi` writer. This format can be identified by a single variable named "data" and a global attribute named ``"awips_id"`` that is set to a string starting with ``"AWIPS_"``. diff --git a/satpy/readers/seviri_base.py b/satpy/readers/seviri_base.py index cb2ec22a50..d0f453e05b 100644 --- a/satpy/readers/seviri_base.py +++ b/satpy/readers/seviri_base.py @@ -26,6 +26,7 @@ from datetime import datetime, timedelta import numpy as np +from numpy.polynomial.chebyshev import Chebyshev import dask.array as da import xarray.ufuncs as xu @@ -288,3 +289,16 @@ def _vis_calibrate(self, data, solar_irradiance): """Calibrate to reflectance.""" return data * 100.0 / solar_irradiance + + +def chebyshev(coefs, time, domain): + """Evaluate a Chebyshev Polynomial + + Args: + coefs (list, np.array): Coefficients defining the polynomial + time (int, float): Time where to evaluate the polynomial + domain (list, tuple): Domain (or time interval) for which the polynomial is defined: [left, right] + + Reference: Appendix A in the MSG Level 1.5 Image Data Format Description. + """ + return Chebyshev(coefs, domain=domain)(time) - 0.5 * coefs[0] diff --git a/satpy/readers/seviri_l1b_hrit.py b/satpy/readers/seviri_l1b_hrit.py index 1e26fad2a4..39ed45eeba 100644 --- a/satpy/readers/seviri_l1b_hrit.py +++ b/satpy/readers/seviri_l1b_hrit.py @@ -37,6 +37,7 @@ from datetime import datetime import numpy as np +import pyproj from pyresample import geometry @@ -46,7 +47,7 @@ annotation_header, base_hdr_map, image_data_function) -from satpy.readers.seviri_base import SEVIRICalibrationHandler +from satpy.readers.seviri_base import SEVIRICalibrationHandler, chebyshev from satpy.readers.seviri_base import (CHANNEL_NAMES, VIS_CHANNELS, CALIB, SATNUM) from satpy.readers.seviri_l1b_native_hdr import (hrit_prologue, hrit_epilogue, @@ -106,6 +107,10 @@ ('fine', 'u1', (3, ))]) +class NoValidNavigationCoefs(Exception): + pass + + class HRITMSGPrologueFileHandler(HRITFileHandler): """SEVIRI HRIT prologue reader. """ @@ -120,6 +125,7 @@ def __init__(self, filename, filename_info, filetype_info, calib_mode='nominal', msg_text_headers)) self.prologue = {} self.read_prologue() + self.satpos = None service = filename_info['service'] if service == '': @@ -141,6 +147,92 @@ def read_prologue(self): else: self.prologue.update(recarray2dict(impf)) + def get_satpos(self): + """Get actual satellite position in geodetic coordinates (WGS-84) + + Returns: Longitude [deg east], Latitude [deg north] and Altitude [m] + """ + if self.satpos is None: + logger.debug("Computing actual satellite position") + + try: + # Get satellite position in cartesian coordinates + x, y, z = self._get_satpos_cart() + + # Transform to geodetic coordinates + geocent = pyproj.Proj(proj='geocent') + a, b = self.get_earth_radii() + latlong = pyproj.Proj(proj='latlong', a=a, b=b, units='meters') + lon, lat, alt = pyproj.transform(geocent, latlong, x, y, z) + except NoValidNavigationCoefs as err: + logger.warning(err) + lon = lat = alt = None + + # Cache results + self.satpos = lon, lat, alt + + return self.satpos + + def _get_satpos_cart(self): + """Determine satellite position in earth-centered cartesion coordinates + + The coordinates as a function of time are encoded in the coefficients of an 8th-order Chebyshev polynomial. + In the prologue there is one set of coefficients for each coordinate (x, y, z). The coordinates are obtained by + evalutaing the polynomials at the start time of the scan. + + Returns: x, y, z [m] + """ + orbit_polynomial = self.prologue['SatelliteStatus']['Orbit']['OrbitPolynomial'] + + # Find Chebyshev coefficients for the given time + coef_idx = self._find_navigation_coefs() + tstart = orbit_polynomial['StartTime'][0, coef_idx] + tend = orbit_polynomial['EndTime'][0, coef_idx] + + # Obtain cartesian coordinates (x, y, z) of the satellite by evaluating the Chebyshev polynomial at the + # start time of the scan. Express timestamps in microseconds since 1970-01-01 00:00. + time = self.prologue['ImageAcquisition']['PlannedAcquisitionTime']['TrueRepeatCycleStart'] + time64 = np.datetime64(time).astype('int64') + domain = [np.datetime64(tstart).astype('int64'), + np.datetime64(tend).astype('int64')] + x = chebyshev(coefs=orbit_polynomial['X'][coef_idx], time=time64, domain=domain) + y = chebyshev(coefs=orbit_polynomial['Y'][coef_idx], time=time64, domain=domain) + z = chebyshev(coefs=orbit_polynomial['Z'][coef_idx], time=time64, domain=domain) + + return x*1000, y*1000, z*1000 # km -> m + + def _find_navigation_coefs(self): + """Find navigation coefficients for the current time + + The navigation Chebyshev coefficients are only valid for a certain time interval. The header entry + SatelliteStatus/Orbit/OrbitPolynomial contains multiple coefficients for multiple time intervals. Find the + coefficients which are valid for the nominal timestamp of the scan. + + Returns: Corresponding index in the coefficient list. + """ + # Find index of interval enclosing the nominal timestamp of the scan + time = np.datetime64(self.prologue['ImageAcquisition']['PlannedAcquisitionTime']['TrueRepeatCycleStart']) + intervals_tstart = self.prologue['SatelliteStatus']['Orbit']['OrbitPolynomial']['StartTime'][0].astype( + 'datetime64') + intervals_tend = self.prologue['SatelliteStatus']['Orbit']['OrbitPolynomial']['EndTime'][0].astype( + 'datetime64') + try: + return np.where(np.logical_and(time >= intervals_tstart, time < intervals_tend))[0][0] + except IndexError: + raise NoValidNavigationCoefs('Unable to find navigation coefficients valid for {}'.format(time)) + + def get_earth_radii(self): + """Get earth radii from prologue + + Returns: + Equatorial radius, polar radius [m] + """ + earth_model = self.prologue['GeometricProcessing']['EarthModel'] + a = earth_model['EquatorialRadius'] * 1000 + b = (earth_model['NorthPolarRadius'] + + earth_model['SouthPolarRadius']) / 2.0 * 1000 + return a, b + class HRITMSGEpilogueFileHandler(HRITFileHandler): """SEVIRI HRIT epilogue reader. @@ -230,6 +322,7 @@ def __init__(self, filename, filename_info, filetype_info, msg_variable_length_headers, msg_text_headers)) + self.prologue_ = prologue self.prologue = prologue.prologue self.epilogue = epilogue.epilogue self._filename_info = filename_info @@ -247,15 +340,26 @@ def _get_header(self): earth_model = self.prologue['GeometricProcessing']['EarthModel'] self.mda['offset_corrected'] = earth_model['TypeOfEarthModel'] == 2 - b = (earth_model['NorthPolarRadius'] + - earth_model['SouthPolarRadius']) / 2.0 * 1000 - self.mda['projection_parameters'][ - 'a'] = earth_model['EquatorialRadius'] * 1000 + + # Projection + a, b = self.prologue_.get_earth_radii() + self.mda['projection_parameters']['a'] = a self.mda['projection_parameters']['b'] = b ssp = self.prologue['ImageDescription'][ 'ProjectionDescription']['LongitudeOfSSP'] self.mda['projection_parameters']['SSP_longitude'] = ssp self.mda['projection_parameters']['SSP_latitude'] = 0.0 + + # Navigation + actual_lon, actual_lat, actual_alt = self.prologue_.get_satpos() + self.mda['navigation_parameters']['satellite_nominal_longitude'] = self.prologue['SatelliteStatus'][ + 'SatelliteDefinition']['NominalLongitude'] + self.mda['navigation_parameters']['satellite_nominal_latitude'] = 0.0 + self.mda['navigation_parameters']['satellite_actual_longitude'] = actual_lon + self.mda['navigation_parameters']['satellite_actual_latitude'] = actual_lat + self.mda['navigation_parameters']['satellite_actual_altitude'] = actual_alt + + # Misc self.platform_id = self.prologue["SatelliteStatus"][ "SatelliteDefinition"]["SatelliteId"] self.platform_name = "Meteosat-" + SATNUM[self.platform_id] @@ -292,7 +396,16 @@ def get_xy_from_linecol(self, line, col, offsets, factors): return x__, y__ def get_area_extent(self, size, offsets, factors, platform_height): - """Get the area extent of the file.""" + """Get the area extent of the file. + + Until December 2017, the data is shifted by 1.5km SSP North and West against the nominal GEOS projection. Since + December 2017 this offset has been corrected. A flag in the data indicates if the correction has been applied. + If no correction was applied, adjust the area extent to match the shifted data. + + For more information see Section 3.1.4.2 in the MSG Level 1.5 Image Data Format Description. The correction + of the area extent is documented in a `developer's memo `_. + """ nlines, ncols = size h = platform_height @@ -312,8 +425,14 @@ def get_area_extent(self, size, offsets, factors, platform_height): np.deg2rad(ur_x) * h, np.deg2rad(ur_y) * h) if not self.mda['offset_corrected']: + # Geo-referencing offset present. Adjust area extent to match the shifted data. Note that we have to adjust + # the corners in the *opposite* direction, i.e. S-E. Think of it as if the coastlines were fixed and you + # dragged the image to S-E until coastlines and data area aligned correctly. + # + # Although the image is flipped upside-down and left-right, the projection coordinates retain their + # properties, i.e. positive x/y is East/North, respectively. xadj = 1500 - yadj = 1500 + yadj = -1500 aex = (aex[0] + xadj, aex[1] + yadj, aex[2] + xadj, aex[3] + yadj) @@ -404,6 +523,12 @@ def get_dataset(self, key, info): res.attrs['satellite_latitude'] = self.mda[ 'projection_parameters']['SSP_latitude'] res.attrs['satellite_altitude'] = self.mda['projection_parameters']['h'] + res.attrs['projection'] = {'satellite_longitude': self.mda['projection_parameters']['SSP_longitude'], + 'satellite_latitude': self.mda['projection_parameters']['SSP_latitude'], + 'satellite_altitude': self.mda['projection_parameters']['h']} + res.attrs['navigation'] = self.mda['navigation_parameters'].copy() + res.attrs['georef_offset_corrected'] = self.mda['offset_corrected'] + return res def calibrate(self, data, calibration): diff --git a/satpy/readers/viirs_edr_active_fires.py b/satpy/readers/viirs_edr_active_fires.py new file mode 100644 index 0000000000..371606be38 --- /dev/null +++ b/satpy/readers/viirs_edr_active_fires.py @@ -0,0 +1,110 @@ +# Copyright (c) 2019 Satpy Developers +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +# +"""VIIRS Active Fires reader +********************* +This module implements readers for VIIRS Active Fires NetCDF and +ASCII files. +""" + +from satpy.readers.netcdf_utils import NetCDF4FileHandler +from satpy.readers.file_handlers import BaseFileHandler +import os +import dask.dataframe as dd +import xarray as xr + + +class VIIRSActiveFiresFileHandler(NetCDF4FileHandler): + """NetCDF4 reader for VIIRS Active Fires + """ + + def get_dataset(self, dsid, dsinfo): + """Get dataset function + + Args: + dsid: Dataset ID + param2: Dataset Information + + Returns: + Dask DataArray: Data + + """ + data = self[dsinfo.get('file_key', dsid.name)] + data.attrs.update(dsinfo) + + data.attrs["platform_name"] = self['/attr/satellite_name'] + data.attrs["sensor"] = self['/attr/instrument_name'] + + return data + + @property + def start_time(self): + return self.filename_info['start_time'] + + @property + def end_time(self): + return self.filename_info.get('end_time', self.start_time) + + @property + def sensor_name(self): + return self["sensor"] + + @property + def platform_name(self): + return self["platform_name"] + + +class VIIRSActiveFiresTextFileHandler(BaseFileHandler): + """ASCII reader for VIIRS Active Fires + """ + def __init__(self, filename, filename_info, filetype_info): + """Makes sure filepath is valid and then reads data into a Dask DataFrame + + Args: + filename: Filename + filename_info: Filename information + filetype_info: Filetype information + """ + super(VIIRSActiveFiresTextFileHandler, self).__init__(filename, filename_info, filetype_info) + + if not os.path.isfile(filename): + return + + self.file_content = dd.read_csv(filename, skiprows=15, header=None, + names=["latitude", "longitude", + "T13", "Along-scan", "Along-track", "detection_confidence", + "power"]) + + def get_dataset(self, dsid, dsinfo): + ds = self[dsid.name].to_dask_array(lengths=True) + data_array = xr.DataArray(ds, dims=("y",), attrs={"platform_name": "unknown", "sensor": "viirs"}) + data_array.attrs.update(dsinfo) + + return data_array + + @property + def start_time(self): + return self.filename_info['start_time'] + + @property + def end_time(self): + return self.filename_info.get('end_time', self.start_time) + + def __getitem__(self, key): + return self.file_content[key] + + def __contains__(self, item): + return item in self.file_content diff --git a/satpy/readers/viirs_sdr.py b/satpy/readers/viirs_sdr.py index ed5fc9385f..64d7219389 100644 --- a/satpy/readers/viirs_sdr.py +++ b/satpy/readers/viirs_sdr.py @@ -124,7 +124,7 @@ def _get_invalid_info(granule_data): class VIIRSSDRFileHandler(HDF5FileHandler): - """VIIRS HDF5 File Reader + """VIIRS HDF5 File Reader. """ def __init__(self, filename, filename_info, filetype_info, use_tc=None, **kwargs): @@ -304,10 +304,16 @@ def concatenate_dataset(self, dataset_group, var_path): else: scan_size = 16 scans_path = 'All_Data/{dataset_group}_All/NumberOfScans' - scans_path = scans_path.format(dataset_group=DATASET_KEYS[dataset_group]) + number_of_granules_path = 'Data_Products/{dataset_group}/{dataset_group}_Aggr/attr/AggregateNumberGranules' + nb_granules_path = number_of_granules_path.format(dataset_group=DATASET_KEYS[dataset_group]) + scans = [] + for granule in range(self[nb_granules_path]): + scans_path = 'Data_Products/{dataset_group}/{dataset_group}_Gran_{granule}/attr/N_Number_Of_Scans' + scans_path = scans_path.format(dataset_group=DATASET_KEYS[dataset_group], granule=granule) + scans.append(self[scans_path]) start_scan = 0 data_chunks = [] - scans = self[scans_path] + scans = xr.DataArray(scans) variable = self[var_path] # check if these are single per-granule value if variable.size != scans.size: @@ -331,6 +337,14 @@ def mask_fill_values(self, data, ds_info): return data.where(data < fill_min) def get_dataset(self, dataset_id, ds_info): + """Get the dataset corresponding to *dataset_id*. + + The size of the return DataArray will be dependent on the number of + scans actually sensed, and not necessarily the regular 768 scanlines + that the file contains for each granule. To that end, the number of + scans for each granule is read from: + `Data_Products/...Gran_x/N_Number_Of_Scans`. + """ dataset_group = [ds_group for ds_group in ds_info['dataset_groups'] if ds_group in self.datasets] if not dataset_group: return diff --git a/satpy/readers/virr_l1b.py b/satpy/readers/virr_l1b.py new file mode 100644 index 0000000000..ddba0545fa --- /dev/null +++ b/satpy/readers/virr_l1b.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# Copyright (c) 2016-2018 Satpy developers +# +# This file is part of satpy. +# +# satpy is free software: you can redistribute it and/or modify it under the +# terms of the GNU General Public License as published by the Free Software +# Foundation, either version 3 of the License, or (at your option) any later +# version. +# +# satpy is distributed in the hope that it will be useful, but WITHOUT ANY +# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +# A PARTICULAR PURPOSE. See the GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along with +# satpy. If not, see . +"""Interface to VIRR (Visible and Infra-Red Radiometer) level 1b format. + +The file format is HDF5. Important attributes: + + - Latitude + - Longitude + - SolarZenith + - EV_Emissive + - EV_RefSB + - Emissive_Radiance_Offsets + - Emissive_Radiance_Scales + - RefSB_Cal_Coefficients + - RefSB_Effective_Wavelength + - Emmisive_Centroid_Wave_Number + +Supported satellites: + + - FY-3B and FY-3C. + +For more information: + + - https://www.wmo-sat.info/oscar/instruments/view/607. + +""" + +from datetime import datetime +from satpy.readers.hdf5_utils import HDF5FileHandler +from pyspectral.blackbody import blackbody_wn_rad2temp as rad2temp +import numpy as np +import xarray as xr +import dask.array as da +import logging + + +class VIRR_L1B(HDF5FileHandler): + """VIRR_L1B reader.""" + + def __init__(self, filename, filename_info, filetype_info): + super(VIRR_L1B, self).__init__(filename, filename_info, filetype_info) + logging.debug('day/night flag for {0}: {1}'.format(filename, self['/attr/Day Or Night Flag'])) + self.geolocation_prefix = filetype_info['geolocation_prefix'] + self.platform_id = filename_info['platform_id'] + self.l1b_prefix = 'Data/' + self.wave_number = 'Emissive_Centroid_Wave_Number' + # Else filename_info['platform_id'] == FY3C. + if filename_info['platform_id'] == 'FY3B': + self.l1b_prefix = '' + self.wave_number = 'Emmisive_Centroid_Wave_Number' + + def get_dataset(self, dataset_id, ds_info): + file_key = self.geolocation_prefix + ds_info.get('file_key', dataset_id.name) + if self.platform_id == 'FY3B': + file_key = file_key.replace('Data/', '') + data = self.get(file_key) + if data is None: + logging.error('File key "{0}" could not be found in file {1}'.format(file_key, self.filename)) + band_index = ds_info.get('band_index') + if band_index is not None: + data = data[band_index] + data = data.where((data >= self[file_key + '/attr/valid_range'][0]) & + (data <= self[file_key + '/attr/valid_range'][1])) + if 'E' in dataset_id.name: + slope = self[self.l1b_prefix + 'Emissive_Radiance_Scales'].data[:, band_index][:, np.newaxis] + intercept = self[self.l1b_prefix + 'Emissive_Radiance_Offsets'].data[:, band_index][:, np.newaxis] + radiance_data = rad2temp(self['/attr/' + self.wave_number][band_index] * 100, + (data * slope + intercept) * 1e-5) + data = xr.DataArray(da.from_array(radiance_data, data.chunks), + coords=data.coords, dims=data.dims, name=data.name, attrs=data.attrs) + elif 'R' in dataset_id.name: + slope = self['/attr/RefSB_Cal_Coefficients'][0::2] + intercept = self['/attr/RefSB_Cal_Coefficients'][1::2] + data = data * slope[band_index] + intercept[band_index] + else: + data = data.where((data >= self[file_key + '/attr/valid_range'][0]) & + (data <= self[file_key + '/attr/valid_range'][1])) + data = self[file_key + '/attr/Intercept'] + self[file_key + '/attr/Slope'] * data + new_dims = {old: new for old, new in zip(data.dims, ('y', 'x'))} + data = data.rename(new_dims) + data.attrs.update({'platform_name': self['/attr/Satellite Name'], + 'sensor': self['/attr/Sensor Identification Code']}) + data.attrs.update(ds_info) + units = self.get(file_key + '/attr/units') + if units is not None and str(units).lower() != 'none': + data.attrs.update({'units': self.get(file_key + '/attr/units')}) + elif data.attrs.get('calibration') == 'reflectance': + data.attrs.update({'units': '%'}) + else: + data.attrs.update({'units': '1'}) + return data + + @property + def start_time(self): + start_time = self['/attr/Observing Beginning Date'] + 'T' + self['/attr/Observing Beginning Time'] + 'Z' + return datetime.strptime(start_time, '%Y-%m-%dT%H:%M:%S.%fZ') + + @property + def end_time(self): + end_time = self['/attr/Observing Ending Date'] + 'T' + self['/attr/Observing Ending Time'] + 'Z' + return datetime.strptime(end_time, '%Y-%m-%dT%H:%M:%S.%fZ') diff --git a/satpy/readers/yaml_reader.py b/satpy/readers/yaml_reader.py index dbbb0d2195..f55d84bc32 100644 --- a/satpy/readers/yaml_reader.py +++ b/satpy/readers/yaml_reader.py @@ -611,7 +611,6 @@ def _load_dataset_data(self, # Update the metadata proj.attrs['start_time'] = file_handlers[0].start_time proj.attrs['end_time'] = file_handlers[-1].end_time - return proj def _preferred_filetype(self, filetypes): diff --git a/satpy/resample.py b/satpy/resample.py index bec93565b9..3b575cd146 100644 --- a/satpy/resample.py +++ b/satpy/resample.py @@ -19,12 +19,12 @@ # # You should have received a copy of the GNU General Public License # along with this program. If not, see . -"""SatPy provides multiple resampling algorithms for resampling geolocated +"""Satpy provides multiple resampling algorithms for resampling geolocated data to uniform projected grids. The easiest way to perform resampling in -SatPy is through the :class:`~satpy.scene.Scene` object's +Satpy is through the :class:`~satpy.scene.Scene` object's :meth:`~satpy.scene.Scene.resample` method. Additional utility functions are also available to assist in resampling data. Below is more information on -resampling with SatPy as well as links to the relevant API documentation for +resampling with Satpy as well as links to the relevant API documentation for available keyword arguments. Resampling algorithms @@ -82,7 +82,7 @@ Caching for geostationary data ------------------------------ -SatPy will do its best to reuse calculations performed to resample datasets, +Satpy will do its best to reuse calculations performed to resample datasets, but it can only do this for the current processing and will lose this information when the process/script ends. Some resampling algorithms, like ``nearest`` and ``bilinear``, can benefit by caching intermediate data on disk in the directory @@ -324,11 +324,18 @@ def precompute(self, mask=None, radius_of_influence=None, epsilon=0, "resampler. Cached parameters are affected by " "masked pixels. Will not cache results.") cache_dir = None - + # TODO: move this to pyresample if radius_of_influence is None: try: radius_of_influence = source_geo_def.lons.resolution * 3 - except (AttributeError, TypeError): + except AttributeError: + try: + radius_of_influence = max(abs(source_geo_def.pixel_size_x), + abs(source_geo_def.pixel_size_y)) * 3 + except AttributeError: + radius_of_influence = 1000 + + except TypeError: radius_of_influence = 10000 kwargs = dict(source_geo_def=source_geo_def, @@ -503,7 +510,7 @@ def precompute(self, cache_dir=None, swath_usage=0, **kwargs): if cache_dir: LOG.warning("'cache_dir' is not used by EWA resampling") - # SatPy/PyResample don't support dynamic grids out of the box yet + # Satpy/PyResample don't support dynamic grids out of the box yet lons, lats = source_geo_def.get_lonlats() if isinstance(lons, xr.DataArray): # get dask arrays diff --git a/satpy/scene.py b/satpy/scene.py index c19ba62b27..6a61c68588 100644 --- a/satpy/scene.py +++ b/satpy/scene.py @@ -146,6 +146,8 @@ def __init__(self, filenames=None, reader=None, filter_parameters=None, reader_k if filter_parameters: if reader_kwargs is None: reader_kwargs = {} + else: + reader_kwargs = reader_kwargs.copy() reader_kwargs.setdefault('filter_parameters', {}).update(filter_parameters) if filenames and isinstance(filenames, str): @@ -586,6 +588,8 @@ def crop(self, area=None, ll_bbox=None, xy_bbox=None, dataset_ids=None): # get the lowest resolution area, use it as the base of the slice # this makes sure that the other areas *should* be a consistent factor min_area = new_scn.min_area() + if isinstance(area, str): + area = get_area_def(area) new_min_area, min_y_slice, min_x_slice = self._slice_area_from_bbox( min_area, area, ll_bbox, xy_bbox) new_target_areas = {} @@ -616,6 +620,51 @@ def crop(self, area=None, ll_bbox=None, xy_bbox=None, dataset_ids=None): return new_scn + def aggregate(self, dataset_ids=None, boundary='exact', side='left', func='mean', **dim_kwargs): + """Create an aggregated version of the Scene. + + Args: + dataset_ids (iterable): DatasetIDs to include in the returned + `Scene`. Defaults to all datasets. + func (string): Function to apply on each aggregation window. One of + 'mean', 'sum', 'min', 'max', 'median', 'argmin', + 'argmax', 'prod', 'std', 'var'. + 'mean' is the default. + boundary: Not implemented. + side: Not implemented. + dim_kwargs: the size of the windows to aggregate. + + Returns: + A new aggregated scene + + See also: + xarray.DataArray.coarsen + + Example: + `scn.aggregate(func='min', x=2, y=2)` will aggregate 2x2 pixels by + applying the `min` function. + """ + new_scn = self.copy(datasets=dataset_ids) + + for src_area, ds_ids in new_scn.iter_by_area(): + if src_area is None: + for ds_id in ds_ids: + new_scn.datasets[ds_id] = self[ds_id] + continue + + if boundary != 'exact': + raise NotImplementedError("boundary modes appart from 'exact' are not implemented yet.") + target_area = src_area.aggregate(**dim_kwargs) + resolution = max(target_area.pixel_size_x, target_area.pixel_size_y) + for ds_id in ds_ids: + res = self[ds_id].coarsen(boundary=boundary, side=side, func=func, **dim_kwargs) + + new_scn.datasets[ds_id] = getattr(res, func)() + new_scn.datasets[ds_id].attrs['area'] = target_area + new_scn.datasets[ds_id].attrs['resolution'] = resolution + + return new_scn + def get(self, key, default=None): """Return value from DatasetDict with optional default.""" return self.datasets.get(key, default) @@ -960,7 +1009,7 @@ def _resampled_scene(self, new_scn, destination_area, reduce_data=True, if reduce_data: key = source_area try: - slices, source_area = reductions[key] + (slice_x, slice_y), source_area = reductions[key] except KeyError: slice_x, slice_y = source_area.get_area_slices(destination_area) source_area = source_area[slice_y, slice_x] @@ -977,8 +1026,7 @@ def _resampled_scene(self, new_scn, destination_area, reduce_data=True, self.resamplers[key] = resampler kwargs = resample_kwargs.copy() kwargs['resampler'] = resamplers[source_area] - res = resample_dataset(dataset, destination_area, - **kwargs) + res = resample_dataset(dataset, destination_area, **kwargs) new_datasets[ds_id] = res if ds_id in new_scn.datasets: new_scn.datasets[ds_id] = res diff --git a/satpy/tests/__init__.py b/satpy/tests/__init__.py index b0d3d462e4..85e85f97c4 100644 --- a/satpy/tests/__init__.py +++ b/satpy/tests/__init__.py @@ -26,7 +26,7 @@ import sys from satpy.tests import (reader_tests, test_dataset, test_file_handlers, - test_readers, test_resample, + test_readers, test_resample, test_demo, test_scene, test_utils, test_writers, test_yaml_reader, writer_tests, test_enhancements, compositor_tests, test_multiscene) @@ -49,6 +49,7 @@ def suite(): mysuite.addTests(test_writers.suite()) mysuite.addTests(test_readers.suite()) mysuite.addTests(test_resample.suite()) + mysuite.addTests(test_demo.suite()) mysuite.addTests(test_yaml_reader.suite()) mysuite.addTests(reader_tests.suite()) mysuite.addTests(writer_tests.suite()) diff --git a/satpy/tests/compositor_tests/__init__.py b/satpy/tests/compositor_tests/__init__.py index 95697d5e0f..0042ea0005 100644 --- a/satpy/tests/compositor_tests/__init__.py +++ b/satpy/tests/compositor_tests/__init__.py @@ -550,6 +550,7 @@ class TestCloudTopHeightCompositor(unittest.TestCase): """Test the CloudTopHeightCompositor.""" def test_call(self): + """Test the CloudTopHeight composite generation.""" from satpy.composites.cloud_products import CloudTopHeightCompositor cmap_comp = CloudTopHeightCompositor('test_cmap_compositor') palette = xr.DataArray(np.array([[0, 0, 0], [127, 127, 127], [255, 255, 255]]), @@ -569,6 +570,34 @@ def test_call(self): np.testing.assert_allclose(res, exp) +class TestPrecipCloudsCompositor(unittest.TestCase): + """Test the PrecipClouds compositor.""" + + def test_call(self): + """Test the precip composite generation.""" + from satpy.composites.cloud_products import PrecipCloudsRGB + cmap_comp = PrecipCloudsRGB('test_precip_compositor') + + data_light = xr.DataArray(np.array([[80, 70, 60, 0], [20, 30, 40, 255]], dtype=np.uint8), + dims=['y', 'x'], attrs={'_FillValue': 255}) + data_moderate = xr.DataArray(np.array([[60, 50, 40, 0], [20, 30, 40, 255]], dtype=np.uint8), + dims=['y', 'x'], attrs={'_FillValue': 255}) + data_intense = xr.DataArray(np.array([[40, 30, 20, 0], [20, 30, 40, 255]], dtype=np.uint8), + dims=['y', 'x'], attrs={'_FillValue': 255}) + data_flags = xr.DataArray(np.array([[0, 0, 4, 0], [0, 0, 0, 0]], dtype=np.uint8), + dims=['y', 'x']) + res = cmap_comp([data_light, data_moderate, data_intense, data_flags]) + + exp = np.array([[[0.24313725, 0.18235294, 0.12156863, np.nan], + [0.12156863, 0.18235294, 0.24313725, np.nan]], + [[0.62184874, 0.51820728, 0.41456583, np.nan], + [0.20728291, 0.31092437, 0.41456583, np.nan]], + [[0.82913165, 0.7254902, 0.62184874, np.nan], + [0.20728291, 0.31092437, 0.41456583, np.nan]]]) + + np.testing.assert_allclose(res, exp) + + class TestGenericCompositor(unittest.TestCase): """Test generic compositor.""" @@ -709,6 +738,7 @@ def suite(): mysuite.addTest(loader.loadTestsFromTestCase(TestPaletteCompositor)) mysuite.addTest(loader.loadTestsFromTestCase(TestCloudTopHeightCompositor)) mysuite.addTest(loader.loadTestsFromTestCase(TestGenericCompositor)) + mysuite.addTest(loader.loadTestsFromTestCase(TestPrecipCloudsCompositor)) return mysuite diff --git a/satpy/tests/reader_tests/__init__.py b/satpy/tests/reader_tests/__init__.py index a4db9605ad..cf58fc37b2 100644 --- a/satpy/tests/reader_tests/__init__.py +++ b/satpy/tests/reader_tests/__init__.py @@ -25,7 +25,7 @@ import sys from satpy.tests.reader_tests import (test_abi_l1b, test_hrit_base, - test_viirs_sdr, test_viirs_l1b, + test_viirs_sdr, test_viirs_l1b, test_virr_l1b, test_seviri_l1b_native, test_seviri_base, test_hdf5_utils, test_netcdf_utils, test_hdf4_utils, test_utils, @@ -38,7 +38,8 @@ test_nc_slstr, test_olci_nc, test_viirs_edr_flood, test_nwcsaf_nc, test_seviri_l1b_hrit, test_sar_c_safe, - test_modis_l1b) + test_modis_l1b, test_viirs_edr_active_fires, + test_safe_sar_l2_ocn) if sys.version_info < (2, 7): @@ -53,6 +54,7 @@ def suite(): mysuite.addTests(test_abi_l1b.suite()) mysuite.addTests(test_viirs_sdr.suite()) mysuite.addTests(test_viirs_l1b.suite()) + mysuite.addTests(test_virr_l1b.suite()) mysuite.addTests(test_hrit_base.suite()) mysuite.addTests(test_seviri_l1b_native.suite()) mysuite.addTests(test_seviri_base.suite()) @@ -82,5 +84,7 @@ def suite(): mysuite.addTests(test_seviri_l1b_hrit.suite()) mysuite.addTests(test_sar_c_safe.suite()) mysuite.addTests(test_modis_l1b.suite()) + mysuite.addTests(test_viirs_edr_active_fires.suite()) + mysuite.addTests(test_safe_sar_l2_ocn.suite()) return mysuite diff --git a/satpy/tests/reader_tests/test_ahi_hsd.py b/satpy/tests/reader_tests/test_ahi_hsd.py index d5e4e7bbea..dce7ecb6b8 100644 --- a/satpy/tests/reader_tests/test_ahi_hsd.py +++ b/satpy/tests/reader_tests/test_ahi_hsd.py @@ -27,6 +27,7 @@ except ImportError: import mock +import warnings import numpy as np import dask.array as da from datetime import datetime @@ -131,6 +132,10 @@ def setUp(self, fromfile, np2str): np2str.side_effect = lambda x: x m = mock.mock_open() with mock.patch('satpy.readers.ahi_hsd.open', m, create=True): + # Check if file handler raises exception for invalid calibration mode + with self.assertRaises(ValueError): + fh = AHIHSDFileHandler(None, {'segment_number': 8, 'total_segments': 10}, None, calib_mode='BAD_MODE') + fh = AHIHSDFileHandler(None, {'segment_number': 8, 'total_segments': 10}, None) fh.proj_info = {'CFAC': 40932549, 'COFF': 5500.5, @@ -175,10 +180,15 @@ def test_time_properties(self): return_value=None) def test_calibrate(self, *mocks): """Test calibration""" + def_cali = [-0.0037, 15.20] + upd_cali = [-0.0074, 30.40] + bad_cali = [0.0, 0.0] fh = AHIHSDFileHandler() + fh.calib_mode = 'NOMINAL' fh._header = { - 'block5': {'gain_count2rad_conversion': [-0.0037], - 'offset_count2rad_conversion': [15.20], + 'block5': {'band_number': [5], + 'gain_count2rad_conversion': [def_cali[0]], + 'offset_count2rad_conversion': [def_cali[1]], 'central_wave_length': [10.4073], }, 'calibration': {'coeff_rad2albedo_conversion': [0.0019255], 'speed_of_light': [299792458.0], @@ -186,7 +196,9 @@ def test_calibrate(self, *mocks): 'boltzmann_constant': [1.3806488e-23], 'c0_rad2tb_conversion': [-0.116127314574], 'c1_rad2tb_conversion': [1.00099153832], - 'c2_rad2tb_conversion': [-1.76961091571e-06]}, + 'c2_rad2tb_conversion': [-1.76961091571e-06], + 'cali_gain_count2rad_conversion': [upd_cali[0]], + 'cali_offset_count2rad_conversion': [upd_cali[1]]}, } # Counts @@ -213,6 +225,35 @@ def test_calibrate(self, *mocks): refl = fh.calibrate(data=counts, calibration='reflectance') self.assertTrue(np.allclose(refl, refl_exp)) + # Updated calibration + # Standard operation + fh.calib_mode = 'UPDATE' + rad_exp = np.array([[30.4, 23.0], + [15.6, 0.]]) + rad = fh.calibrate(data=counts, calibration='radiance') + self.assertTrue(np.allclose(rad, rad_exp)) + + # Case for no updated calibration available (older data) + fh._header = { + 'block5': {'band_number': [5], + 'gain_count2rad_conversion': [def_cali[0]], + 'offset_count2rad_conversion': [def_cali[1]], + 'central_wave_length': [10.4073], }, + 'calibration': {'coeff_rad2albedo_conversion': [0.0019255], + 'speed_of_light': [299792458.0], + 'planck_constant': [6.62606957e-34], + 'boltzmann_constant': [1.3806488e-23], + 'c0_rad2tb_conversion': [-0.116127314574], + 'c1_rad2tb_conversion': [1.00099153832], + 'c2_rad2tb_conversion': [-1.76961091571e-06], + 'cali_gain_count2rad_conversion': [bad_cali[0]], + 'cali_offset_count2rad_conversion': [bad_cali[1]]}, + } + rad = fh.calibrate(data=counts, calibration='radiance') + rad_exp = np.array([[15.2, 11.5], + [7.8, 0]]) + self.assertTrue(np.allclose(rad, rad_exp)) + @mock.patch('satpy.readers.ahi_hsd.AHIHSDFileHandler._read_header') @mock.patch('satpy.readers.ahi_hsd.AHIHSDFileHandler._read_data') @mock.patch('satpy.readers.ahi_hsd.AHIHSDFileHandler._mask_invalid') @@ -236,6 +277,50 @@ def test_read_band(self, calibrate, *mocks): ref_mask = np.logical_not(get_geostationary_mask(self.fh.area).compute()) self.assertTrue(np.all(mask == ref_mask)) + # Test if masking space pixels disables with appropriate flag + self.fh.mask_space = False + with mock.patch('satpy.readers.ahi_hsd.AHIHSDFileHandler._mask_space') as mask_space: + self.fh.read_band(info=mock.MagicMock(), key=mock.MagicMock()) + mask_space.assert_not_called() + + def test_blocklen_error(self, *mocks): + open_name = '%s.open' % __name__ + fpos = 50 + with mock.patch(open_name, create=True) as mock_open: + with mock_open(mock.MagicMock(), 'r') as fp_: + # Expected and actual blocklength match + fp_.tell.return_value = 50 + with warnings.catch_warnings(record=True) as w: + self.fh._check_fpos(fp_, fpos, 0, 'header 1') + self.assertTrue(len(w) == 0) + + # Expected and actual blocklength do not match + fp_.tell.return_value = 100 + with warnings.catch_warnings(record=True) as w: + self.fh._check_fpos(fp_, fpos, 0, 'header 1') + self.assertTrue(len(w) > 0) + + @mock.patch('satpy.readers.ahi_hsd.AHIHSDFileHandler._check_fpos') + def test_read_header(self, *mocks): + nhdr = [ + {'blocklength': 0}, + {'blocklength': 0}, + {'blocklength': 0}, + {'blocklength': 0}, + {'blocklength': 0, 'band_number': [4]}, + {'blocklength': 0}, + {'blocklength': 0}, + {'blocklength': 0}, + {'blocklength': 0, 'numof_correction_info_data': [1]}, + {'blocklength': 0}, + {'blocklength': 0, 'number_of_observation_times': [1]}, + {'blocklength': 0}, + {'blocklength': 0, 'number_of_error_info_data': [1]}, + {'blocklength': 0}, + {'blocklength': 0}] + with mock.patch('numpy.fromfile', side_effect=nhdr): + self.fh._read_header(mock.MagicMock()) + def suite(): """The test suite for test_scene.""" diff --git a/satpy/tests/reader_tests/test_safe_sar_l2_ocn.py b/satpy/tests/reader_tests/test_safe_sar_l2_ocn.py new file mode 100644 index 0000000000..5eafa2a029 --- /dev/null +++ b/satpy/tests/reader_tests/test_safe_sar_l2_ocn.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- + +# Copyright (c) 2019 Pytroll developers + +# Author(s): + +# Trygve Aspenes + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +"""Module for testing the satpy.readers.safe_sar_l2_ocn module. +""" +import sys +import numpy as np +import xarray as xr + +from satpy import DatasetID + +if sys.version_info < (2, 7): + import unittest2 as unittest +else: + import unittest + +try: + import unittest.mock as mock +except ImportError: + import mock + + +class TestSAFENC(unittest.TestCase): + """Test various SAFE SAR L2 OCN file handlers.""" + @mock.patch('satpy.readers.safe_sar_l2_ocn.xr') + @mock.patch.multiple('satpy.readers.safe_sar_l2_ocn.SAFENC', + __abstractmethods__=set()) + def setUp(self, xr_): + from satpy.readers.safe_sar_l2_ocn import SAFENC + + self.channels = ['owiWindSpeed', 'owiLon', 'owiLat', 'owiHs', 'owiNrcs', 'foo', + 'owiPolarisationName', 'owiCalConstObsi'] + # Mock file access to return a fake dataset. + self.dummy3d = np.zeros((2, 2, 1)) + self.dummy2d = np.zeros((2, 2)) + self.dummy1d = np.zeros((2)) + self.band = 1 + self.nc = xr.Dataset( + {'owiWindSpeed': xr.DataArray(self.dummy2d, dims=('owiAzSize', 'owiRaSize'), attrs={'_FillValue': np.nan}), + 'owiLon': xr.DataArray(data=self.dummy2d, dims=('owiAzSize', 'owiRaSize')), + 'owiLat': xr.DataArray(data=self.dummy2d, dims=('owiAzSize', 'owiRaSize')), + 'owiHs': xr.DataArray(data=self.dummy3d, dims=('owiAzSize', 'owiRaSize', 'oswPartition')), + 'owiNrcs': xr.DataArray(data=self.dummy3d, dims=('owiAzSize', 'owiRaSize', 'oswPolarization')), + 'foo': xr.DataArray(self.dummy2d, dims=('owiAzSize', 'owiRaSize')), + 'owiPolarisationName': xr.DataArray(self.dummy1d, dims=('owiPolarisation')), + 'owiCalConstObsi': xr.DataArray(self.dummy1d, dims=('owiIncSize')) + }, + attrs={'_FillValue': np.nan, + 'missionName': 'S1A'}) + xr_.open_dataset.return_value = self.nc + + # Instantiate reader using the mocked open_dataset() method. Also, make + # the reader believe all abstract methods have been implemented. + self.reader = SAFENC(filename='dummy', + filename_info={'start_time': 0, + 'end_time': 0, + 'fstart_time': 0, + 'fend_time': 0, + 'polarization': 'vv'}, + filetype_info={}) + + def test_init(self): + """Tests reader initialization""" + self.assertEqual(self.reader.start_time, 0) + self.assertEqual(self.reader.end_time, 0) + self.assertEqual(self.reader.fstart_time, 0) + self.assertEqual(self.reader.fend_time, 0) + + def test_get_dataset(self): + for ch in self.channels: + dt = self.reader.get_dataset( + key=DatasetID(name=ch), info={}) + # ... this only compares the valid (unmasked) elements + self.assertTrue(np.all(self.nc[ch] == dt.to_masked_array()), + msg='get_dataset() returns invalid data for ' + 'dataset {}'.format(ch)) + +# @mock.patch('xarray.open_dataset') +# def test_init(self, mocked_dataset): +# """Test basic init with no extra parameters.""" +# from satpy.readers.safe_sar_l2_ocn import SAFENC +# from satpy import DatasetID +# +# print(mocked_dataset) +# ds_id = DatasetID(name='foo') +# filename_info = {'mission_id': 'S3A', 'product_type': 'foo', +# 'start_time': 0, 'end_time': 0, +# 'fstart_time': 0, 'fend_time': 0, +# 'polarization': 'vv'} +# +# test = SAFENC('S1A_IW_OCN__2SDV_20190228T075834_20190228T075849_026127_02EA43_8846.SAFE/measurement/' +# 's1a-iw-ocn-vv-20190228t075741-20190228t075800-026127-02EA43-001.nc', filename_info, 'c') +# print(test) +# mocked_dataset.assert_called() +# test.get_dataset(ds_id, filename_info) + + +def suite(): + """The test suite for test_safe_sar_l2_ocn.""" + loader = unittest.TestLoader() + mysuite = unittest.TestSuite() + mysuite.addTest(loader.loadTestsFromTestCase(TestSAFENC)) + return mysuite + + +if __name__ == '__main__': + unittest.main() diff --git a/satpy/tests/reader_tests/test_seviri_base.py b/satpy/tests/reader_tests/test_seviri_base.py index df1e0a77c5..c12c60c43f 100644 --- a/satpy/tests/reader_tests/test_seviri_base.py +++ b/satpy/tests/reader_tests/test_seviri_base.py @@ -25,7 +25,7 @@ import sys import numpy as np -from satpy.readers.seviri_base import dec10216 +from satpy.readers.seviri_base import dec10216, chebyshev if sys.version_info < (2, 7): import unittest2 as unittest @@ -45,9 +45,27 @@ def test_dec10216(self): self.assertTrue(np.all(res == exp)) +class TestChebyshev(unittest.TestCase): + def chebyshev4(self, c, x, domain): + """Evaluate 4th order Chebyshev polynomial""" + start_x, end_x = domain + t = (x - 0.5 * (end_x + start_x)) / (0.5 * (end_x - start_x)) + return c[0] + c[1]*t + c[2]*(2*t**2 - 1) + c[3]*(4*t**3 - 3*t) - 0.5*c[0] + + def test_chebyshev(self): + coefs = [1, 2, 3, 4] + time = 123 + domain = [120, 130] + res = chebyshev(coefs=[1, 2, 3, 4], time=time, domain=domain) + exp = self.chebyshev4(coefs, time, domain) + self.assertTrue(np.allclose(res, exp)) + + def suite(): """The test suite for test_scene.""" loader = unittest.TestLoader() mysuite = unittest.TestSuite() - mysuite.addTest(loader.loadTestsFromTestCase(TestDec10216)) + tests = [TestDec10216, TestChebyshev] + for test in tests: + mysuite.addTest(loader.loadTestsFromTestCase(test)) return mysuite diff --git a/satpy/tests/reader_tests/test_seviri_l1b_hrit.py b/satpy/tests/reader_tests/test_seviri_l1b_hrit.py index ab110542ef..1e8cfc6bef 100644 --- a/satpy/tests/reader_tests/test_seviri_l1b_hrit.py +++ b/satpy/tests/reader_tests/test_seviri_l1b_hrit.py @@ -27,8 +27,8 @@ import numpy as np import xarray as xr -from satpy.readers.seviri_l1b_hrit import (HRITMSGFileHandler, HRITMSGPrologueFileHandler, - HRITMSGEpilogueFileHandler) +from satpy.readers.seviri_l1b_hrit import (HRITMSGFileHandler, HRITMSGPrologueFileHandler, HRITMSGEpilogueFileHandler, + NoValidNavigationCoefs) from satpy.readers.seviri_base import CHANNEL_NAMES, VIS_CHANNELS from satpy.dataset import DatasetID @@ -51,6 +51,7 @@ def new_get_hd(instance, hdr_info): 'b': 6356583.80, 'h': 35785831.00, 'SSP_longitude': 0.0} + instance.mda['navigation_parameters'] = {} instance.mda['total_header_length'] = 12 @@ -69,12 +70,16 @@ def setUp(self, fromfile): with mock.patch.object(HRITMSGFileHandler, '_get_hd', new=new_get_hd): newopen.return_value.__enter__.return_value.tell.return_value = 1 prologue = mock.MagicMock() - prologue.prologue = {"SatelliteStatus": {"SatelliteDefinition": {"SatelliteId": 324}}, + prologue.prologue = {"SatelliteStatus": {"SatelliteDefinition": {"SatelliteId": 324, + "NominalLongitude": 47}}, 'GeometricProcessing': {'EarthModel': {'TypeOfEarthModel': 2, 'NorthPolarRadius': 10, 'SouthPolarRadius': 10, 'EquatorialRadius': 10}}, 'ImageDescription': {'ProjectionDescription': {'LongitudeOfSSP': 0.0}}} + prologue.get_satpos.return_value = None, None, None + prologue.get_earth_radii.return_value = None, None + self.reader = HRITMSGFileHandler( 'filename', {'platform_shortname': 'MSG3', @@ -99,6 +104,13 @@ def setUp(self, fromfile): self.reader.mda['projection_parameters']['b'] = 6356583.8 self.reader.mda['projection_parameters']['h'] = 35785831.0 self.reader.mda['projection_parameters']['SSP_longitude'] = 44 + self.reader.mda['projection_parameters']['SSP_latitude'] = 0.0 + self.reader.mda['navigation_parameters'] = {} + self.reader.mda['navigation_parameters']['satellite_nominal_longitude'] = 47 + self.reader.mda['navigation_parameters']['satellite_nominal_latitude'] = 0.0 + self.reader.mda['navigation_parameters']['satellite_actual_longitude'] = 47.5 + self.reader.mda['navigation_parameters']['satellite_actual_latitude'] = -0.5 + self.reader.mda['navigation_parameters']['satellite_actual_altitude'] = 35783328 def test_get_xy_from_linecol(self): """Test get_xy_from_linecol.""" @@ -130,6 +142,13 @@ def test_get_area_def(self): (-77771774058.38356, -3720765401003.719, 30310525626438.438, 77771774058.38356)) + # Data shifted by 1.5km to N-W + self.reader.mda['offset_corrected'] = False + area = self.reader.get_area_def(DatasetID('VIS006')) + self.assertEqual(area.area_extent, + (-77771772558.38356, -3720765402503.719, + 30310525627938.438, 77771772558.38356)) + @mock.patch('satpy.readers.hrit_base.np.memmap') def test_read_band(self, memmap): nbits = self.reader.mda['number_of_bits_per_pixel'] @@ -220,10 +239,82 @@ def get_header_patched(self): self.assertRaises(ValueError, HRITMSGFileHandler, filename=None, filename_info=None, filetype_info=None, prologue=pro, epilogue=epi, calib_mode='invalid') + @mock.patch('satpy.readers.seviri_l1b_hrit.HRITFileHandler.get_dataset') + @mock.patch('satpy.readers.seviri_l1b_hrit.HRITMSGFileHandler.calibrate') + def test_get_dataset(self, calibrate, parent_get_dataset): + key = mock.MagicMock(calibration='calibration') + info = {'units': 'units', 'wavelength': 'wavelength', 'standard_name': 'standard_name'} + parent_get_dataset.return_value = mock.MagicMock() + calibrate.return_value = mock.MagicMock(attrs={}) + + res = self.reader.get_dataset(key, info) + + # Test method calls + parent_get_dataset.assert_called_with(key, info) + calibrate.assert_called_with(parent_get_dataset(), key.calibration) + + # Test attributes + attrs_exp = info.copy() + attrs_exp.update({ + 'platform_name': self.reader.platform_name, + 'sensor': 'seviri', + 'satellite_longitude': self.reader.mda['projection_parameters']['SSP_longitude'], + 'satellite_latitude': self.reader.mda['projection_parameters']['SSP_latitude'], + 'satellite_altitude': self.reader.mda['projection_parameters']['h'], + 'projection': {'satellite_longitude': self.reader.mda['projection_parameters']['SSP_longitude'], + 'satellite_latitude': self.reader.mda['projection_parameters']['SSP_latitude'], + 'satellite_altitude': self.reader.mda['projection_parameters']['h']}, + 'navigation': self.reader.mda['navigation_parameters'], + 'georef_offset_corrected': self.reader.mda['offset_corrected'] + }) + + self.assertDictEqual(attrs_exp, res.attrs) + class TestHRITMSGPrologueFileHandler(unittest.TestCase): """Test the HRIT prologue file handler.""" + @mock.patch('satpy.readers.seviri_l1b_hrit.HRITMSGPrologueFileHandler.__init__', return_value=None) + def setUp(self, *mocks): + self.reader = HRITMSGPrologueFileHandler() + self.reader.satpos = None + self.reader.prologue = { + 'GeometricProcessing': { + 'EarthModel': { + 'EquatorialRadius': 6378.169, + 'NorthPolarRadius': 6356.5838, + 'SouthPolarRadius': 6356.5838 + } + }, + 'ImageAcquisition': { + 'PlannedAcquisitionTime': { + 'TrueRepeatCycleStart': datetime(2006, 1, 1, 12, 15, 9, 304888) + } + }, + 'SatelliteStatus': { + 'Orbit': { + 'OrbitPolynomial': { + 'StartTime': np.array([ + [datetime(2006, 1, 1, 6), datetime(2006, 1, 1, 12), datetime(2006, 1, 1, 18)]]), + 'EndTime': np.array([ + [datetime(2006, 1, 1, 12), datetime(2006, 1, 1, 18), datetime(2006, 1, 2, 0)]]), + 'X': [np.zeros(8), + [8.41607082e+04, 2.94319260e+00, 9.86748617e-01, -2.70135453e-01, + -3.84364650e-02, 8.48718433e-03, 7.70548174e-04, -1.44262718e-04], + np.zeros(8)], + 'Y': [np.zeros(8), + [-5.21170255e+03, 5.12998948e+00, -1.33370453e+00, -3.09634144e-01, + 6.18232793e-02, 7.50505681e-03, -1.35131011e-03, -1.12054405e-04], + np.zeros(8)], + 'Z': [np.zeros(8), + [-6.51293855e+02, 1.45830459e+02, 5.61379400e+01, -3.90970565e+00, + -7.38137565e-01, 3.06131644e-02, 3.82892428e-03, -1.12739309e-04], + np.zeros(8)], + } + } + } + } + @mock.patch('satpy.readers.seviri_l1b_hrit.HRITMSGPrologueFileHandler.read_prologue') @mock.patch('satpy.readers.hrit_base.HRITFileHandler.__init__', autospec=True) def test_calibrate(self, init, *mocks): @@ -238,6 +329,51 @@ def init_patched(self, *args, **kwargs): ext_calib_coefs={}, calib_mode='nominal') + def test_find_navigation_coefs(self): + """Test identification of navigation coefficients""" + + self.assertEqual(self.reader._find_navigation_coefs(), 1) + + # No interval enclosing the given timestamp + self.reader.prologue['ImageAcquisition']['PlannedAcquisitionTime'][ + 'TrueRepeatCycleStart'] = datetime(2000, 1, 1) + self.assertRaises(NoValidNavigationCoefs, self.reader._find_navigation_coefs) + + @mock.patch('satpy.readers.seviri_l1b_hrit.HRITMSGPrologueFileHandler._find_navigation_coefs') + def test_get_satpos_cart(self, find_navigation_coefs): + """Test satellite position in cartesian coordinates""" + find_navigation_coefs.return_value = 1 + x, y, z = self.reader._get_satpos_cart() + self.assertTrue(np.allclose([x, y, z], [42078421.37095518, -2611352.744615312, -419828.9699940758])) + + @mock.patch('satpy.readers.seviri_l1b_hrit.HRITMSGPrologueFileHandler._get_satpos_cart') + def test_get_satpos(self, get_satpos_cart): + """Test satellite position in spherical coordinates""" + get_satpos_cart.return_value = [42078421.37095518, -2611352.744615312, -419828.9699940758] + lon, lat, dist = self.reader.get_satpos() + self.assertTrue(np.allclose(lon, lat, dist), [-3.5511754052132387, -0.5711189258409902, 35783328.146167226]) + + # Test cache + self.reader.get_satpos() + self.assertEqual(get_satpos_cart.call_count, 1) + + # No valid coefficients + self.reader.satpos = None # reset cache + get_satpos_cart.side_effect = NoValidNavigationCoefs + self.reader.prologue['ImageAcquisition']['PlannedAcquisitionTime'][ + 'TrueRepeatCycleStart'] = datetime(2000, 1, 1) + self.assertTupleEqual(self.reader.get_satpos(), (None, None, None)) + + def test_get_earth_radii(self): + """Test readout of earth radii""" + earth_model = self.reader.prologue['GeometricProcessing']['EarthModel'] + earth_model['EquatorialRadius'] = 2 + earth_model['NorthPolarRadius'] = 1 + earth_model['SouthPolarRadius'] = 2 + a, b = self.reader.get_earth_radii() + self.assertEqual(a, 2000) + self.assertEqual(b, 1500) + class TestHRITMSGEpilogueFileHandler(unittest.TestCase): """Test the HRIT epilogue file handler.""" diff --git a/satpy/tests/reader_tests/test_viirs_edr_active_fires.py b/satpy/tests/reader_tests/test_viirs_edr_active_fires.py new file mode 100644 index 0000000000..fc77cd7575 --- /dev/null +++ b/satpy/tests/reader_tests/test_viirs_edr_active_fires.py @@ -0,0 +1,223 @@ +# Copyright (c) 2019 Satpy Developers +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +# +# +"""VIIRS Active Fires Tests +********************* +This module implements tests for VIIRS Active Fires NetCDF and ASCII file +readers. +""" + +import sys +import os +import numpy as np +import io +import dask.dataframe as dd +import pandas as pd +from satpy.tests.reader_tests.test_netcdf_utils import FakeNetCDF4FileHandler +from satpy.readers.file_handlers import BaseFileHandler +if sys.version_info < (2, 7): + import unittest2 as unittest +else: + import unittest + +try: + from unittest import mock +except ImportError: + import mock + + +DEFAULT_FILE_SHAPE = (1, 100) + +DEFAULT_LATLON_FILE_DTYPE = np.float32 +DEFAULT_LATLON_FILE_DATA = np.arange(start=43, stop=45, step=0.02, + dtype=DEFAULT_LATLON_FILE_DTYPE).reshape(DEFAULT_FILE_SHAPE) + +DEFAULT_DETECTION_FILE_DTYPE = np.ubyte +DEFAULT_DETECTION_FILE_DATA = np.arange(start=60, stop=100, step=0.4, + dtype=DEFAULT_DETECTION_FILE_DTYPE).reshape(DEFAULT_FILE_SHAPE) + +DEFAULT_M13_FILE_DTYPE = np.float32 +DEFAULT_M13_FILE_DATA = np.arange(start=300, stop=340, step=0.4, + dtype=DEFAULT_M13_FILE_DTYPE).reshape(DEFAULT_FILE_SHAPE) + +DEFAULT_POWER_FILE_DTYPE = np.float32 +DEFAULT_POWER_FILE_DATA = np.arange(start=1, stop=25, step=0.24, + dtype=DEFAULT_POWER_FILE_DTYPE).reshape(DEFAULT_FILE_SHAPE) + + +class FakeFiresNetCDF4FileHandler(FakeNetCDF4FileHandler): + """Swap in CDF4 file handler""" + def get_test_content(self, filename, filename_info, filename_type): + """Mimic reader input file content""" + file_content = {} + file_content['/attr/satellite_name'] = "AFEDR_npp" + file_content['/attr/instrument_name'] = 'VIIRS' + + file_content['Fire Pixels/FP_latitude'] = DEFAULT_LATLON_FILE_DATA + file_content['Fire Pixels/FP_longitude'] = DEFAULT_LATLON_FILE_DATA + file_content['Fire Pixels/FP_power'] = DEFAULT_POWER_FILE_DATA + file_content['Fire Pixels/FP_T13'] = DEFAULT_M13_FILE_DATA + file_content['Fire Pixels/FP_confidence'] = DEFAULT_DETECTION_FILE_DATA + file_content['Fire Pixels/attr/units'] = 'none' + file_content['Fire Pixels/shape'] = DEFAULT_FILE_SHAPE + + # convert to xarrays + from xarray import DataArray + for key, val in file_content.items(): + if isinstance(val, np.ndarray): + attrs = {} + for a in ['FP_latitude', 'FP_longitude', 'FP_T13', 'FP_confidence']: + if key + '/attr/' + a in file_content: + attrs[a] = file_content[key + '/attr/' + a] + if val.ndim > 1: + file_content[key] = DataArray(val, dims=('fakeDim0', 'fakeDim1'), attrs=attrs) + else: + file_content[key] = DataArray(val, attrs=attrs) + + return file_content + + +class FakeFiresTextFileHandler(BaseFileHandler): + def __init__(self, filename, filename_info, filetype_info, **kwargs): + """Get fake file content from 'get_test_content'""" + super(FakeFiresTextFileHandler, self).__init__(filename, filename_info, filetype_info) + self.file_content = self.get_test_content() + + def get_test_content(self): + fake_file = io.StringIO(u'''\n\n\n\n\n\n\n\n\n\n\n\n\n\n + 24.64015007, -107.57017517, 317.38290405, 0.75, 0.75, 40, 4.28618050 + 25.90660477, -100.06127167, 331.17962646, 0.75, 0.75, 81, 20.61096764''') + + return dd.from_pandas(pd.read_csv(fake_file, skiprows=15, header=None, + names=["latitude", "longitude", + "T13", "Along-scan", "Along-track", + "detection_confidence", + "power"]), chunksize=1) + + +class TestVIIRSActiveFiresNetCDF4(unittest.TestCase): + """Test VIIRS Fires Reader""" + yaml_file = 'viirs_edr_active_fires.yaml' + + def setUp(self): + """Wrap CDF4 file handler with own fake file handler""" + from satpy.config import config_search_paths + from satpy.readers.viirs_edr_active_fires import VIIRSActiveFiresFileHandler + self.reader_configs = config_search_paths(os.path.join('readers', self.yaml_file)) + self.p = mock.patch.object(VIIRSActiveFiresFileHandler, '__bases__', (FakeFiresNetCDF4FileHandler,)) + self.fake_handler = self.p.start() + self.p.is_local = True + + def tearDown(self): + """Stop wrapping the CDF4 file handler""" + self.p.stop() + + def test_init(self): + """Test basic init with no extra parameters""" + from satpy.readers import load_reader + r = load_reader(self.reader_configs) + loadables = r.select_files_from_pathnames([ + 'AFEDR_npp_d20180829_t2015451_e2017093_b35434_c20180829210527716708_cspp_dev.nc' + ]) + self.assertTrue(len(loadables), 1) + r.create_filehandlers(loadables) + self.assertTrue(r.file_handlers) + + def test_load_dataset(self): + """Test loading all datasets""" + from satpy.readers import load_reader + r = load_reader(self.reader_configs) + loadables = r.select_files_from_pathnames([ + 'AFEDR_npp_d20180829_t2015451_e2017093_b35434_c20180829210527716708_cspp_dev.nc' + ]) + r.create_filehandlers(loadables) + datasets = r.load(['detection_confidence']) + self.assertEqual(len(datasets), 1) + for v in datasets.values(): + self.assertEqual(v.attrs['units'], '%') + + datasets = r.load(['T13']) + self.assertEqual(len(datasets), 1) + for v in datasets.values(): + self.assertEqual(v.attrs['units'], 'K') + + datasets = r.load(['power']) + self.assertEqual(len(datasets), 1) + for v in datasets.values(): + self.assertEqual(v.attrs['units'], 'MW') + + +class TestVIIRSActiveFiresText(unittest.TestCase): + """Test VIIRS Fires Reader""" + yaml_file = 'viirs_edr_active_fires.yaml' + + def setUp(self): + """Wrap file handler with own fake file handler""" + from satpy.config import config_search_paths + from satpy.readers.viirs_edr_active_fires import VIIRSActiveFiresTextFileHandler + self.reader_configs = config_search_paths(os.path.join('readers', self.yaml_file)) + self.p = mock.patch.object(VIIRSActiveFiresTextFileHandler, '__bases__', (FakeFiresTextFileHandler,)) + self.fake_handler = self.p.start() + self.p.is_local = True + + def tearDown(self): + """Stop wrapping the text file handler""" + self.p.stop() + + def test_init(self): + """Test basic init with no extra parameters""" + from satpy.readers import load_reader + r = load_reader(self.reader_configs) + loadables = r.select_files_from_pathnames([ + 'AFEDR_npp_d20180829_t2015451_e2017093_b35434_c20180829210527716708_cspp_dev.txt' + ]) + self.assertTrue(len(loadables), 1) + r.create_filehandlers(loadables) + self.assertTrue(r.file_handlers) + + def test_load_dataset(self): + """Test loading all datasets""" + from satpy.readers import load_reader + r = load_reader(self.reader_configs) + loadables = r.select_files_from_pathnames([ + 'AFEDR_npp_d20180829_t2015451_e2017093_b35434_c20180829210527716708_cspp_dev.txt' + ]) + r.create_filehandlers(loadables) + datasets = r.load(['detection_confidence']) + self.assertEqual(len(datasets), 1) + for v in datasets.values(): + self.assertEqual(v.attrs['units'], '%') + + datasets = r.load(['T13']) + self.assertEqual(len(datasets), 1) + for v in datasets.values(): + self.assertEqual(v.attrs['units'], 'K') + + datasets = r.load(['power']) + self.assertEqual(len(datasets), 1) + for v in datasets.values(): + self.assertEqual(v.attrs['units'], 'MW') + + +def suite(): + """The test suite for testing viirs active fires + """ + loader = unittest.TestLoader() + mysuite = unittest.TestSuite() + mysuite.addTest(loader.loadTestsFromTestCase(TestVIIRSActiveFiresNetCDF4)) + mysuite.addTest(loader.loadTestsFromTestCase(TestVIIRSActiveFiresText)) + + return mysuite diff --git a/satpy/tests/reader_tests/test_viirs_sdr.py b/satpy/tests/reader_tests/test_viirs_sdr.py index c5f9004f4b..d32dc0a212 100644 --- a/satpy/tests/reader_tests/test_viirs_sdr.py +++ b/satpy/tests/reader_tests/test_viirs_sdr.py @@ -78,6 +78,7 @@ def get_test_content(self, filename, filename_info, filetype_info): prefix1 = 'Data_Products/{dataset_group}'.format(dataset_group=dataset_group) prefix2 = '{prefix}/{dataset_group}_Aggr'.format(prefix=prefix1, dataset_group=dataset_group) prefix3 = 'All_Data/{dataset_group}_All'.format(dataset_group=dataset_group) + prefix4 = '{prefix}/{dataset_group}_Gran_0'.format(prefix=prefix1, dataset_group=dataset_group) begin_date = start_time.strftime('%Y%m%d') begin_time = start_time.strftime('%H%M%S.%fZ') ending_date = end_time.strftime('%Y%m%d') @@ -89,7 +90,8 @@ def get_test_content(self, filename, filename_info, filetype_info): else: geo_prefix = None file_content = { - "{prefix3}/NumberOfScans": np.array([48]), + "{prefix2}/attr/AggregateNumberGranules": 1, + "{prefix4}/attr/N_Number_Of_Scans": 48, "{prefix2}/attr/AggregateBeginningDate": begin_date, "{prefix2}/attr/AggregateBeginningTime": begin_time, "{prefix2}/attr/AggregateEndingDate": ending_date, @@ -104,7 +106,7 @@ def get_test_content(self, filename, filename_info, filetype_info): if geo_prefix: file_content['/attr/N_GEO_Ref'] = geo_prefix + filename[5:] for k, v in list(file_content.items()): - file_content[k.format(prefix1=prefix1, prefix2=prefix2, prefix3=prefix3)] = v + file_content[k.format(prefix1=prefix1, prefix2=prefix2, prefix3=prefix3, prefix4=prefix4)] = v if filename[:3] in ['SVM', 'SVI', 'SVD']: if filename[2:5] in ['M{:02d}'.format(x) for x in range(12)] + ['I01', 'I02', 'I03']: diff --git a/satpy/tests/reader_tests/test_virr_l1b.py b/satpy/tests/reader_tests/test_virr_l1b.py new file mode 100644 index 0000000000..8c130b1f6d --- /dev/null +++ b/satpy/tests/reader_tests/test_virr_l1b.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# Copyright (c) 2016-2018 Satpy developers +# +# This file is part of satpy. +# +# satpy is free software: you can redistribute it and/or modify it under the +# terms of the GNU General Public License as published by the Free Software +# Foundation, either version 3 of the License, or (at your option) any later +# version. +# +# satpy is distributed in the hope that it will be useful, but WITHOUT ANY +# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR +# A PARTICULAR PURPOSE. See the GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along with +# satpy. If not, see . +"""Test for readers/virr_l1b.py. +""" +from satpy.tests.reader_tests.test_hdf5_utils import FakeHDF5FileHandler +import sys +import numpy as np +import dask.array as da +import xarray as xr +import os + +if sys.version_info < (2, 7): + import unittest2 as unittest +else: + import unittest + +try: + from unittest import mock +except ImportError: + import mock + + +class FakeHDF5FileHandler2(FakeHDF5FileHandler): + """Swap-in HDF5 File Handler.""" + + def make_test_data(self, dims): + return xr.DataArray(da.from_array(np.ones([dim for dim in dims], dtype=np.float32) * 10, [dim for dim in dims])) + + def _make_file(self, platform_id, geolocation_prefix, l1b_prefix, ECWN, Emissive_units): + dim_0 = 19 + dim_1 = 20 + test_file = { + # Satellite data. + '/attr/Day Or Night Flag': 'D', '/attr/Observing Beginning Date': '2018-12-25', + '/attr/Observing Beginning Time': '21:41:47.090', '/attr/Observing Ending Date': '2018-12-25', + '/attr/Observing Ending Time': '21:47:28.254', '/attr/Satellite Name': platform_id, + '/attr/Sensor Identification Code': 'VIRR', + # Emissive data. + l1b_prefix + 'EV_Emissive': self.make_test_data([3, dim_0, dim_1]), + l1b_prefix + 'EV_Emissive/attr/valid_range': [0, 50000], + l1b_prefix + 'Emissive_Radiance_Scales': self.make_test_data([dim_0, dim_1]), + l1b_prefix + 'EV_Emissive/attr/units': Emissive_units, + l1b_prefix + 'Emissive_Radiance_Offsets': self.make_test_data([dim_0, dim_1]), + '/attr/' + ECWN: [2610.31, 917.6268, 836.2546], + # Reflectance data. + l1b_prefix + 'EV_RefSB': self.make_test_data([7, dim_0, dim_1]), + l1b_prefix + 'EV_RefSB/attr/valid_range': [0, 32767], l1b_prefix + 'EV_RefSB/attr/units': 'none', + '/attr/RefSB_Cal_Coefficients': np.ones(14, dtype=np.float32) * 2 + } + for attribute in ['Latitude', 'Longitude', geolocation_prefix + 'SolarZenith', + geolocation_prefix + 'SensorZenith', geolocation_prefix + 'SolarAzimuth', + geolocation_prefix + 'SensorAzimuth']: + test_file[attribute] = self.make_test_data([dim_0, dim_1]) + test_file[attribute + '/attr/Intercept'] = 0. + test_file[attribute + '/attr/units'] = 'degrees' + if 'Solar' in attribute or 'Sensor' in attribute: + test_file[attribute + '/attr/Slope'] = .01 + if 'Azimuth' in attribute: + test_file[attribute + '/attr/valid_range'] = [0, 18000] + else: + test_file[attribute + '/attr/valid_range'] = [-18000, 18000] + else: + test_file[attribute + '/attr/Slope'] = 1. + if 'Longitude' == attribute: + test_file[attribute + '/attr/valid_range'] = [-180., 180.] + else: + test_file[attribute + '/attr/valid_range'] = [-90., 90.] + return test_file + + def get_test_content(self, filename, filename_info, filetype_info): + """Mimic reader input file content.""" + if filename_info['platform_id'] == 'FY3B': + return self._make_file('FY3B', '', '', 'Emmisive_Centroid_Wave_Number', 'milliWstts/m^2/cm^(-1)/steradian') + return self._make_file(filename_info['platform_id'], 'Geolocation/', 'Data/', + 'Emissive_Centroid_Wave_Number', 'none') + + +class TestVIRRL1BReader(unittest.TestCase): + """Test VIRR L1B Reader.""" + yaml_file = "virr_l1b.yaml" + + def setUp(self): + """Wrap HDF5 file handler with our own fake handler.""" + from satpy.readers.virr_l1b import VIRR_L1B + from satpy.config import config_search_paths + self.reader_configs = config_search_paths(os.path.join('readers', self.yaml_file)) + # http://stackoverflow.com/questions/12219967/how-to-mock-a-base-class-with-python-mock-library + self.p = mock.patch.object(VIRR_L1B, '__bases__', (FakeHDF5FileHandler2,)) + self.fake_handler = self.p.start() + self.p.is_local = True + + def tearDown(self): + """Stop wrapping the HDF5 file handler.""" + self.p.stop() + + def _band_helper(self, attributes, units, calibration, standard_name, + file_type, band_index_size, resolution, level): + self.assertEqual(units, attributes['units']) + self.assertEqual(calibration, attributes['calibration']) + self.assertEqual(standard_name, attributes['standard_name']) + self.assertEqual(file_type, attributes['file_type']) + self.assertTrue(attributes['band_index'] in range(band_index_size)) + self.assertEqual(resolution, attributes['resolution']) + self.assertEqual(level, attributes['level']) + self.assertEqual(('longitude', 'latitude'), attributes['coordinates']) + + def _fy3_helper(self, platform_name, reader, Emissive_units): + import datetime + band_values = {'R1': 22.0, 'R2': 22.0, 'R3': 22.0, 'R4': 22.0, 'R5': 22.0, 'R6': 22.0, 'R7': 22.0, + 'E1': 496.542155, 'E2': 297.444511, 'E3': 288.956557, 'solar_zenith_angle': .1, + 'satellite_zenith_angle': .1, 'solar_azimuth_angle': .1, 'satellite_azimuth_angle': .1, + 'longitude': 10} + datasets = reader.load([band for band, val in band_values.items()]) + for dataset in datasets: + ds = datasets[dataset.name] + attributes = ds.attrs + self.assertEqual('VIRR', attributes['sensor']) + self.assertEqual(platform_name, attributes['platform_name']) + self.assertEqual(datetime.datetime(2018, 12, 25, 21, 41, 47, 90000), attributes['start_time']) + self.assertEqual(datetime.datetime(2018, 12, 25, 21, 47, 28, 254000), attributes['end_time']) + self.assertEqual((19, 20), datasets[dataset.name].shape) + self.assertEqual(('y', 'x'), datasets[dataset.name].dims) + if 'R' in dataset.name: + self._band_helper(attributes, '%', 'reflectance', 'toa_bidirectional_reflectance', 'virr_l1b', 7, 1000, + 1) + elif 'E' in dataset.name: + self._band_helper(attributes, Emissive_units, 'brightness_temperature', + 'toa_brightness_temperature', 'virr_l1b', 3, 1000, 1) + elif dataset.name in ['longitude', 'latitude']: + self.assertEqual('degrees', attributes['units']) + self.assertTrue(attributes['standard_name'] in ['longitude', 'latitude']) + self.assertEqual(['virr_l1b', 'virr_geoxx'], attributes['file_type']) + self.assertEqual(1000, attributes['resolution']) + else: + self.assertEqual('degrees', attributes['units']) + self.assertTrue( + attributes['standard_name'] in ['solar_zenith_angle', 'sensor_zenith_angle', 'solar_azimuth_angle', + 'sensor_azimuth_angle']) + self.assertEqual(['virr_geoxx', 'virr_l1b'], attributes['file_type']) + self.assertEqual(('longitude', 'latitude'), attributes['coordinates']) + self.assertEqual(band_values[dataset.name], + round(float(np.array(ds[ds.shape[0] // 2][ds.shape[1] // 2])), 6)) + + def test_fy3b_file(self): + from satpy.readers import load_reader + FY3B_reader = load_reader(self.reader_configs) + FY3B_file = FY3B_reader.select_files_from_pathnames(['tf2018359214943.FY3B-L_VIRRX_L1B.HDF']) + self.assertTrue(1, len(FY3B_file)) + FY3B_reader.create_filehandlers(FY3B_file) + # Make sure we have some files + self.assertTrue(FY3B_reader.file_handlers) + self._fy3_helper('FY3B', FY3B_reader, 'milliWstts/m^2/cm^(-1)/steradian') + + def test_FY3C_file(self): + from satpy.readers import load_reader + FY3C_reader = load_reader(self.reader_configs) + FY3C_files = FY3C_reader.select_files_from_pathnames(['tf2018359143912.FY3C-L_VIRRX_GEOXX.HDF', + 'tf2018359143912.FY3C-L_VIRRX_L1B.HDF']) + self.assertTrue(2, len(FY3C_files)) + FY3C_reader.create_filehandlers(FY3C_files) + # Make sure we have some files + self.assertTrue(FY3C_reader.file_handlers) + self._fy3_helper('FY3C', FY3C_reader, '1') + + +def suite(): + """The test suite for test_virr_l1b.""" + loader = unittest.TestLoader() + mysuite = unittest.TestSuite() + mysuite.addTest(loader.loadTestsFromTestCase(TestVIRRL1BReader)) + return mysuite diff --git a/satpy/tests/test_dataset.py b/satpy/tests/test_dataset.py index f50e54b9f8..d2df27762e 100644 --- a/satpy/tests/test_dataset.py +++ b/satpy/tests/test_dataset.py @@ -1,7 +1,7 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- # -# Copyright (c) 2015-2018 SatPy Developers +# Copyright (c) 2015-2018 Satpy Developers # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/satpy/tests/test_demo.py b/satpy/tests/test_demo.py new file mode 100644 index 0000000000..bfe567bec3 --- /dev/null +++ b/satpy/tests/test_demo.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# +# Copyright (c) 2019 Satpy developers +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +"""Tests for the satpy.demo module.""" + +import os +import sys + +if sys.version_info < (2, 7): + import unittest2 as unittest +else: + import unittest +try: + from unittest import mock +except ImportError: + import mock + + +class TestDemo(unittest.TestCase): + """Test demo data download functions.""" + + @mock.patch('satpy.demo.google_cloud_platform.gcsfs') + def test_get_us_midlatitude_cyclone_abi(self, gcsfs_mod): + """Test data download function.""" + from satpy.demo import get_us_midlatitude_cyclone_abi + gcsfs_mod.GCSFileSystem = mock.MagicMock() + gcsfs_inst = mock.MagicMock() + gcsfs_mod.GCSFileSystem.return_value = gcsfs_inst + gcsfs_inst.glob.return_value = ['a.nc', 'b.nc'] + self.assertRaises(AssertionError, get_us_midlatitude_cyclone_abi) + self.assertRaises(NotImplementedError, get_us_midlatitude_cyclone_abi, method='unknown') + + gcsfs_inst.glob.return_value = ['a.nc'] * 16 + filenames = get_us_midlatitude_cyclone_abi() + expected = os.path.join('.', 'abi_l1b', '20190314_us_midlatitude_cyclone', 'a.nc') + for fn in filenames: + self.assertEqual(expected, fn) + + +class TestGCPUtils(unittest.TestCase): + """Test Google Cloud Platform utilities.""" + + @mock.patch('satpy.demo.google_cloud_platform.urlopen') + def test_is_gcp_instance(self, uo): + """Test is_google_cloud_instance.""" + from satpy.demo.google_cloud_platform import is_google_cloud_instance, URLError + uo.side_effect = URLError("Test Environment") + self.assertFalse(is_google_cloud_instance()) + + @mock.patch('satpy.demo.google_cloud_platform.gcsfs') + def test_get_bucket_files(self, gcsfs_mod): + """Test get_bucket_files basic cases.""" + from satpy.demo.google_cloud_platform import get_bucket_files + gcsfs_mod.GCSFileSystem = mock.MagicMock() + gcsfs_inst = mock.MagicMock() + gcsfs_mod.GCSFileSystem.return_value = gcsfs_inst + gcsfs_inst.glob.return_value = ['a.nc', 'b.nc'] + filenames = get_bucket_files('*.nc', '.') + expected = [os.path.join('.', 'a.nc'), os.path.join('.', 'b.nc')] + self.assertEqual(expected, filenames) + + gcsfs_inst.glob.return_value = ['a.nc', 'b.nc'] + self.assertRaises(OSError, get_bucket_files, '*.nc', 'does_not_exist') + + # touch the file + open('a.nc', 'w').close() + gcsfs_inst.get.reset_mock() + gcsfs_inst.glob.return_value = ['a.nc'] + filenames = get_bucket_files('*.nc', '.') + self.assertEqual([os.path.join('.', 'a.nc')], filenames) + gcsfs_inst.get.assert_not_called() + + # force redownload + gcsfs_inst.get.reset_mock() + gcsfs_inst.glob.return_value = ['a.nc'] + filenames = get_bucket_files('*.nc', '.', force=True) + self.assertEqual([os.path.join('.', 'a.nc')], filenames) + gcsfs_inst.get.assert_called_once() + + # if we don't get any results then we expect an exception + gcsfs_inst.get.reset_mock() + gcsfs_inst.glob.return_value = [] + self.assertRaises(OSError, get_bucket_files, '*.nc', '.') + + @mock.patch('satpy.demo.google_cloud_platform.gcsfs', None) + def test_no_gcsfs(self): + """Test that 'gcsfs' is required.""" + from satpy.demo.google_cloud_platform import get_bucket_files + self.assertRaises(RuntimeError, get_bucket_files, '*.nc', '.') + + +def suite(): + """The test suite for test_demo.""" + loader = unittest.TestLoader() + mysuite = unittest.TestSuite() + mysuite.addTest(loader.loadTestsFromTestCase(TestDemo)) + mysuite.addTest(loader.loadTestsFromTestCase(TestGCPUtils)) + return mysuite + + +if __name__ == "__main__": + unittest.main() diff --git a/satpy/tests/test_multiscene.py b/satpy/tests/test_multiscene.py index 7184292004..316f747dbd 100644 --- a/satpy/tests/test_multiscene.py +++ b/satpy/tests/test_multiscene.py @@ -257,6 +257,69 @@ def test_save_mp4_distributed(self): self.assertEqual(filenames[1], 'test_save_mp4_ds2_20180101_00_20180102_12.mp4') self.assertEqual(filenames[2], 'test_save_mp4_ds3_20180102_00_20180102_12.mp4') + # Test no distributed client found + mscn = MultiScene(scenes) + fn = os.path.join( + self.base_dir, + 'test_save_mp4_{name}_{start_time:%Y%m%d_%H}_{end_time:%Y%m%d_%H}.mp4') + writer_mock = mock.MagicMock() + client_mock = mock.MagicMock() + client_mock.compute.side_effect = lambda x: tuple(v.compute() for v in x) + client_mock.gather.side_effect = lambda x: x + with mock.patch('satpy.multiscene.imageio.get_writer') as get_writer, \ + mock.patch('satpy.multiscene.get_client', mock.Mock(side_effect=ValueError("No client"))): + get_writer.return_value = writer_mock + # force order of datasets by specifying them + mscn.save_animation(fn, datasets=['ds1', 'ds2', 'ds3']) + + # 2 saves for the first scene + 1 black frame + # 3 for the second scene + self.assertEqual(writer_mock.append_data.call_count, 3 + 3) + filenames = [os.path.basename(args[0][0]) for args in get_writer.call_args_list] + self.assertEqual(filenames[0], 'test_save_mp4_ds1_20180101_00_20180102_12.mp4') + self.assertEqual(filenames[1], 'test_save_mp4_ds2_20180101_00_20180102_12.mp4') + self.assertEqual(filenames[2], 'test_save_mp4_ds3_20180102_00_20180102_12.mp4') + + @mock.patch('satpy.multiscene.get_enhanced_image', _fake_get_enhanced_image) + def test_save_mp4_no_distributed(self): + """Save a series of fake scenes to an mp4 video when distributed isn't available.""" + from satpy import MultiScene + area = _create_test_area() + scenes = _create_test_scenes(area=area) + + # Add a dataset to only one of the Scenes + scenes[1]['ds3'] = _create_test_dataset('ds3') + # Add a start and end time + for ds_id in ['ds1', 'ds2', 'ds3']: + scenes[1][ds_id].attrs['start_time'] = datetime(2018, 1, 2) + scenes[1][ds_id].attrs['end_time'] = datetime(2018, 1, 2, 12) + if ds_id == 'ds3': + continue + scenes[0][ds_id].attrs['start_time'] = datetime(2018, 1, 1) + scenes[0][ds_id].attrs['end_time'] = datetime(2018, 1, 1, 12) + + mscn = MultiScene(scenes) + fn = os.path.join( + self.base_dir, + 'test_save_mp4_{name}_{start_time:%Y%m%d_%H}_{end_time:%Y%m%d_%H}.mp4') + writer_mock = mock.MagicMock() + client_mock = mock.MagicMock() + client_mock.compute.side_effect = lambda x: tuple(v.compute() for v in x) + client_mock.gather.side_effect = lambda x: x + with mock.patch('satpy.multiscene.imageio.get_writer') as get_writer, \ + mock.patch('satpy.multiscene.get_client', None): + get_writer.return_value = writer_mock + # force order of datasets by specifying them + mscn.save_animation(fn, datasets=['ds1', 'ds2', 'ds3']) + + # 2 saves for the first scene + 1 black frame + # 3 for the second scene + self.assertEqual(writer_mock.append_data.call_count, 3 + 3) + filenames = [os.path.basename(args[0][0]) for args in get_writer.call_args_list] + self.assertEqual(filenames[0], 'test_save_mp4_ds1_20180101_00_20180102_12.mp4') + self.assertEqual(filenames[1], 'test_save_mp4_ds2_20180101_00_20180102_12.mp4') + self.assertEqual(filenames[2], 'test_save_mp4_ds3_20180102_00_20180102_12.mp4') + @mock.patch('satpy.multiscene.get_enhanced_image', _fake_get_enhanced_image) def test_save_datasets_simple(self): """Save a series of fake scenes to an PNG images.""" diff --git a/satpy/tests/test_scene.py b/satpy/tests/test_scene.py index 313e650f13..17475aae93 100644 --- a/satpy/tests/test_scene.py +++ b/satpy/tests/test_scene.py @@ -86,6 +86,26 @@ def test_start_end_times(self): self.assertEqual(scene.start_time, r.start_time) self.assertEqual(scene.end_time, r.end_time) + def test_init_preserve_reader_kwargs(self): + import satpy.scene + from satpy.tests.utils import create_fake_reader + from datetime import datetime + with mock.patch('satpy.scene.Scene.create_reader_instances') as cri: + r = create_fake_reader('fake_reader', + start_time=datetime(2017, 1, 1, 0, 0, 0), + end_time=datetime(2017, 1, 1, 1, 0, 0), + ) + cri.return_value = {'fake_reader': r} + reader_kwargs = {'calibration_type': 'gsics'} + scene = satpy.scene.Scene(filenames=['bla'], + base_dir='bli', + sensor='fake_sensor', + filter_parameters={'area': 'euron1'}, + reader_kwargs=reader_kwargs) + self.assertIsNot(reader_kwargs, cri.call_args[1]['reader_kwargs']) + self.assertEqual(scene.start_time, r.start_time) + self.assertEqual(scene.end_time, r.end_time) + def test_init_alone(self): from satpy.scene import Scene from satpy.config import PACKAGE_CONFIG_PATH @@ -168,6 +188,36 @@ def test_create_reader_instances_with_reader(self): reader_kwargs=None, ) + def test_create_reader_instances_with_reader_kwargs(self): + import satpy.scene + from satpy.tests.utils import create_fake_reader + from datetime import datetime + filenames = ["1", "2", "3"] + reader_kwargs = {'calibration_type': 'gsics'} + filter_parameters = {'area': 'euron1'} + reader_kwargs2 = {'calibration_type': 'gsics', 'filter_parameters': filter_parameters} + + with mock.patch('satpy.readers.load_reader') as lr_mock: + r = create_fake_reader('fake_reader', + start_time=datetime(2017, 1, 1, 0, 0, 0), + end_time=datetime(2017, 1, 1, 1, 0, 0), + ) + lr_mock.return_value = r + r.select_path_from_pathnames.return_value = filenames + scene = satpy.scene.Scene(filenames=['bla'], + base_dir='bli', + sensor='fake_sensor', + filter_parameters={'area': 'euron1'}, + reader_kwargs=reader_kwargs) + del scene + self.assertDictEqual(reader_kwargs, r.create_filehandlers.call_args[1]['fh_kwargs']) + scene = satpy.scene.Scene(filenames=['bla'], + base_dir='bli', + sensor='fake_sensor', + reader_kwargs=reader_kwargs2) + self.assertDictEqual(reader_kwargs, r.create_filehandlers.call_args[1]['fh_kwargs']) + del scene + def test_iter(self): from satpy import Scene from xarray import DataArray @@ -448,6 +498,44 @@ def test_crop_rgb(self): self.assertTupleEqual(new_scn1['1'].shape, (3, 184, 714)) self.assertTupleEqual(new_scn1['2'].shape, (92, 3, 357)) + def test_aggregate(self): + """Test the aggregate method.""" + if (sys.version_info < (3, 0)): + self.skipTest("Not implemented in python 2 (xarray).") + from satpy import Scene + from xarray import DataArray + from pyresample.geometry import AreaDefinition + import numpy as np + scene1 = Scene() + area_extent = (-5570248.477339745, -5561247.267842293, 5567248.074173927, + 5570248.477339745) + proj_dict = {'a': 6378169.0, 'b': 6356583.8, 'h': 35785831.0, + 'lon_0': 0.0, 'proj': 'geos', 'units': 'm'} + x_size = 3712 + y_size = 3712 + area_def = AreaDefinition( + 'test', + 'test', + 'test', + proj_dict, + x_size, + y_size, + area_extent, + ) + + scene1["1"] = DataArray(np.ones((y_size, x_size))) + scene1["2"] = DataArray(np.ones((y_size, x_size)), dims=('y', 'x')) + scene1["3"] = DataArray(np.ones((y_size, x_size)), dims=('y', 'x'), + attrs={'area': area_def}) + + scene2 = scene1.aggregate(func='sum', x=2, y=2) + self.assertIs(scene1['1'], scene2['1']) + self.assertIs(scene1['2'], scene2['2']) + np.testing.assert_allclose(scene2['3'].data, 4) + self.assertTupleEqual(scene2['1'].shape, (y_size, x_size)) + self.assertTupleEqual(scene2['2'].shape, (y_size, x_size)) + self.assertTupleEqual(scene2['3'].shape, (y_size / 2, x_size / 2)) + def test_contains(self): from satpy import Scene from xarray import DataArray @@ -1520,18 +1608,16 @@ def test_load_too_many(self, cri, cl): class TestSceneResampling(unittest.TestCase): - """Test resampling a Scene to another Scene object.""" def _fake_resample_dataset(self, dataset, dest_area, **kwargs): """Return copy of dataset pretending it was resampled.""" return dataset.copy() - @mock.patch('satpy.scene.Scene._slice_data') @mock.patch('satpy.scene.resample_dataset') @mock.patch('satpy.composites.CompositorLoader.load_compositors') @mock.patch('satpy.scene.Scene.create_reader_instances') - def test_resample_scene_copy(self, cri, cl, rs, slice_data): + def test_resample_scene_copy(self, cri, cl, rs): """Test that the Scene is properly copied during resampling. The Scene that is created as a copy of the original Scene should not @@ -1554,7 +1640,6 @@ def test_resample_scene_copy(self, cri, cl, rs, slice_data): '+units=m +no_defs') area_def = AreaDefinition('test', 'test', 'test', proj_dict, 5, 5, (-1000., -1500., 1000., 1500.)) area_def.get_area_slices = mock.MagicMock() - get_area_slices = area_def.get_area_slices scene = satpy.scene.Scene(filenames=['bla'], base_dir='bli', reader='fake_reader') @@ -1584,20 +1669,70 @@ def test_resample_scene_copy(self, cri, cl, rs, slice_data): self.assertTupleEqual(tuple(loaded_ids[0]), tuple(DatasetID(name='comp19'))) self.assertTupleEqual(tuple(loaded_ids[1]), tuple(DatasetID(name='new_ds'))) + @mock.patch('satpy.scene.resample_dataset') + @mock.patch('satpy.composites.CompositorLoader.load_compositors') + @mock.patch('satpy.scene.Scene.create_reader_instances') + def test_resample_reduce_data_toggle(self, cri, cl, rs): + """Test that the Scene can be reduced or not reduced during resampling.""" + import satpy.scene + from satpy.tests.utils import create_fake_reader, test_composites + from satpy import DatasetID + from pyresample.geometry import AreaDefinition + from pyresample.utils import proj4_str_to_dict + import dask.array as da + import xarray as xr + cri.return_value = {'fake_reader': create_fake_reader( + 'fake_reader', 'fake_sensor')} + comps, mods = test_composites('fake_sensor') + cl.return_value = (comps, mods) + rs.side_effect = self._fake_resample_dataset + + proj_dict = proj4_str_to_dict('+proj=lcc +datum=WGS84 +ellps=WGS84 ' + '+lon_0=-95. +lat_0=25 +lat_1=25 ' + '+units=m +no_defs') + target_area = AreaDefinition('test', 'test', 'test', proj_dict, 4, 4, (-1000., -1500., 1000., 1500.)) + area_def = AreaDefinition('test', 'test', 'test', proj_dict, 5, 5, (-1000., -1500., 1000., 1500.)) + area_def.get_area_slices = mock.MagicMock() + get_area_slices = area_def.get_area_slices + get_area_slices.return_value = (slice(0, 3, None), slice(0, 3, None)) + area_def_big = AreaDefinition('test', 'test', 'test', proj_dict, 10, 10, (-1000., -1500., 1000., 1500.)) + area_def_big.get_area_slices = mock.MagicMock() + get_area_slices_big = area_def_big.get_area_slices + get_area_slices_big.return_value = (slice(0, 6, None), slice(0, 6, None)) + # Test that data reduction can be disabled scene = satpy.scene.Scene(filenames=['bla'], base_dir='bli', reader='fake_reader') scene.load(['comp19']) scene['comp19'].attrs['area'] = area_def - get_area_slices.return_value = (slice(0, 5, None), slice(0, 5, None)) - scene.resample(area_def, reduce_data=False) - self.assertFalse(slice_data.called) - self.assertFalse(get_area_slices.called) - scene.resample(area_def) - self.assertTrue(slice_data.called_once) - self.assertTrue(get_area_slices.called_once) - scene.resample(area_def, reduce_data=True) - self.assertEqual(slice_data.call_count, 2) - self.assertTrue(get_area_slices.called_once) + scene['comp19_big'] = xr.DataArray( + da.zeros((10, 10)), dims=('y', 'x'), + attrs=scene['comp19'].attrs.copy()) + scene['comp19_big'].attrs['area'] = area_def_big + scene['comp19_copy'] = scene['comp19'].copy() + orig_slice_data = scene._slice_data + # we force the below order of processing to test that success isn't + # based on data of the same resolution being processed together + test_order = [ + DatasetID.from_dict(scene['comp19'].attrs), + DatasetID.from_dict(scene['comp19_big'].attrs), + DatasetID.from_dict(scene['comp19_copy'].attrs), + ] + with mock.patch('satpy.scene.Scene._slice_data') as slice_data, \ + mock.patch('satpy.dataset.dataset_walker') as ds_walker: + ds_walker.return_value = test_order + slice_data.side_effect = orig_slice_data + scene.resample(target_area, reduce_data=False) + self.assertFalse(slice_data.called) + self.assertFalse(get_area_slices.called) + scene.resample(target_area) + self.assertTrue(slice_data.called_once) + self.assertTrue(get_area_slices.called_once) + scene.resample(target_area, reduce_data=True) + # 2 times for each dataset + # once for default (reduce_data=True) + # once for kwarg forced to `True` + self.assertEqual(slice_data.call_count, 2 * 3) + self.assertTrue(get_area_slices.called_once) @mock.patch('satpy.composites.CompositorLoader.load_compositors') @mock.patch('satpy.scene.Scene.create_reader_instances') diff --git a/satpy/tests/writer_tests/test_scmi.py b/satpy/tests/writer_tests/test_scmi.py index 28442a2176..f90e8e5db4 100644 --- a/satpy/tests/writer_tests/test_scmi.py +++ b/satpy/tests/writer_tests/test_scmi.py @@ -1,7 +1,7 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- # -# Copyright (c) 2017-2018 SatPy Developers +# Copyright (c) 2017-2018 Satpy Developers # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by diff --git a/satpy/writers/__init__.py b/satpy/writers/__init__.py index 7931e73cc6..d8b59b1735 100644 --- a/satpy/writers/__init__.py +++ b/satpy/writers/__init__.py @@ -184,8 +184,9 @@ def _determine_mode(dataset): def add_overlay(orig, area, coast_dir, color=(0, 0, 0), width=0.5, resolution=None, - level_coast=1, level_borders=1, fill_value=None): - """Add coastline and political borders to image. + level_coast=1, level_borders=1, fill_value=None, + grid=None): + """Add coastline, political borders and grid(graticules) to image. Uses ``color`` for feature colors where ``color`` is a 3-element tuple of integers between 0 and 255 representing (R, G, B). @@ -204,6 +205,20 @@ def add_overlay(orig, area, coast_dir, color=(0, 0, 0), width=0.5, resolution=No | 'c' | Crude resolution | 25 km | +-----+-------------------------+---------+ + ``grid`` is a dictionary with key values as documented in detail in pycoast + + eg. overlay={'grid': {'major_lonlat': (10, 10), + 'write_text': False, + 'outline': (224, 224, 224), + 'width': 0.5}} + + Here major_lonlat is plotted every 10 deg for both longitude and latitude, + no labels for the grid lines are plotted, the color used for the grid lines + is light gray, and the width of the gratucules is 0.5 pixels. + + For grid if aggdraw is used, font option is mandatory, if not write_text is set to False + eg. font = aggdraw.Font('black', '/usr/share/fonts/truetype/msttcorefonts/Arial.ttf', + opacity=127, size=16) """ if area is None: @@ -249,6 +264,12 @@ def add_overlay(orig, area, coast_dir, color=(0, 0, 0), width=0.5, resolution=No resolution=resolution, width=width, level=level_coast) cw_.add_borders(img, area, outline=color, resolution=resolution, width=width, level=level_borders) + # Only add grid if major_lonlat is given. + if grid and 'major_lonlat' in grid and grid['major_lonlat']: + major_lonlat = grid.pop('major_lonlat') + minor_lonlat = grid.pop('minor_lonlat', major_lonlat) + + cw_.add_grid(img, area, major_lonlat, minor_lonlat, **grid) arr = da.from_array(np.array(img) / 255.0, chunks=CHUNK_SIZE) @@ -440,6 +461,19 @@ def show(dataset, **kwargs): def to_image(dataset): + """convert ``dataset`` into a :class:`~trollimage.xrimage.XRImage` instance. + + Convert the ``dataset`` into an instance of the + :class:`~trollimage.xrimage.XRImage` class. This function makes no other + changes. To get an enhanced image, possibly with overlays and decoration, + see :func:`~get_enhanced_image`. + + Args: + dataset (xarray.DataArray): Data to be converted to an image. + + Returns: + Instance of :class:`~trollimage.xrimage.XRImage`. + """ dataset = dataset.squeeze() if dataset.ndim < 2: raise ValueError("Need at least a 2D array to make an image.") diff --git a/satpy/writers/scmi.py b/satpy/writers/scmi.py index 61b687049e..70840a52eb 100644 --- a/satpy/writers/scmi.py +++ b/satpy/writers/scmi.py @@ -672,7 +672,7 @@ def set_global_attrs(self, physical_element, awips_id, sector_id, self.nc.Conventions = "CF-1.7" if creator is None: from satpy import __version__ - self.nc.creator = "SatPy Version {} - SCMI Writer".format(__version__) + self.nc.creator = "Satpy Version {} - SCMI Writer".format(__version__) else: self.nc.creator = creator self.nc.creation_time = datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S') diff --git a/setup.py b/setup.py index a89eb10822..0073472a0e 100644 --- a/setup.py +++ b/setup.py @@ -33,7 +33,7 @@ requires = ['numpy >=1.13', 'pillow', 'pyresample >=1.10.3', 'trollsift', 'trollimage >=1.5.1', 'pykdtree', 'six', 'pyyaml', 'xarray >=0.10.1', - 'dask[array] >=0.17.1'] + 'dask[array] >=0.17.1', 'pyproj'] test_requires = ['behave', 'h5py', 'netCDF4', 'pyhdf', 'imageio', 'libtiff', 'rasterio', 'geoviews']