"Manual" application of corrections

[marfel]

Is your feature request related to a problem? Please describe.
Maybe I'm missing something here, in which case I apologize. But it seems to me I cannot simply

1. load one or several MSG channels,
2. remap to my desired projection,
3. apply the SZA correction (and/or other corrections) and
4. save the channel as a dataset.

Describe the solution you'd like
See above.

Describe any changes to existing user workflow
I don't know satpy well enough to comment on that.

Additional python or other dependencies
No.

Describe any changes required to the build process
I think none.

Describe alternatives you've considered
Right now, as a workaround I load a composite the prerequisites of which include the desired corrections, then I resample with generate=False, unload=False. Then I save the corrected and remapped datasets. This seems pretty whacky...
I could perhaps also write a compositor which does nothing except the corrections, but again this would be unclean because I am not aiming for a composite, simply for corrected channels.

Additional context
A potential performance issue with the current workaround might be that the corrections right now are applied during load, i.e. on the full scene, even if my target area is very small.

[mraspaud]

I thought this was possible, but apparently it's broken right now. In principle, you would need to create a specific DatasetID instance to load your data. Eg:

from satpy import DatasetID
dsid = DatasetID(name='VIS006', modifiers=('sunz_corrected'))
from satpy import Scene
from glob import glob
scn = Scene(sensor="seviri",
                filenames=glob(
                     "/home/a001673/data/satellite/Meteosat-11/seviri/lvl1.5/2018/02/28/HRIT/*"),
                reader='hrit_msg',
                )
scn.load(dsid)

@djhoese any clue why this crashes ?

[mraspaud]

@marfel also, since version 0.9.0 we will be using dask arrays (lazy computation) so the corrections will not be computed everywhere in principle. You can try out the alpha (almost beta) 0.9.0 by cloning the master branch from this github repo.

[marfel]

Thanks! In fact I already did that. Right now learning a lot about dask and xarray...

[djhoese]

@mraspaud How does it crash? Traceback? Also why do you provide 'sensor' to the Scene?

[djhoese]

WFM with ABI L1B C01 doing:

from satpy import Scene, DatasetID
from glob import glob
filenames = glob('/data/data/abi/20170901/*.nc')
scn = Scene(reader='abi_l1b', filenames=filenames)
scn.load([DatasetID(name='C01', modifiers=('sunz_corrected',))])
scn['C01'].compute()

And yes the sunz corrector is executed.

[mraspaud]

No luck with the hrit_msg reader

[DEBUG: 2018-06-14 15:25:40 : satpy.composites] Looking for composites config file seviri.yaml
[DEBUG: 2018-06-14 15:25:40 : satpy.composites] Looking for composites config file visir.yaml
Traceback (most recent call last):
  File "test_hrit_seviri.py", line 92, in <module>
    main()
  File "test_hrit_seviri.py", line 61, in main
    scn.load([composite])
  File "/home/a001673/usr/src/satpy/satpy/scene.py", line 696, in load
    raise KeyError("Unknown datasets: {}".format(unknown_str))
KeyError: "Unknown datasets: DatasetID(name='VIS006', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers='')"

[marfel]

I'm looking at native_msg (slightly modified for HRV channel, should not matter here):

[DEBUG: 2018-06-14 16:25:29 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/geo/MSG/config'
[DEBUG: 2018-06-14 16:25:29 : satpy.readers] Reading ['/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/etc/readers/native_msg.yaml', '/geo/MSG/config/readers/native_msg.yaml', '/geo/MSG/config/readers/native_msg.yaml']
[DEBUG: 2018-06-14 16:25:29 : satpy.readers.yaml_reader] Assigning to native_msg: ['/geo/MSG/offline/MSG3-SEV/2015/08/01/MSG3-SEVI-MSG15-0100-NA-20150801154240.668000000Z-1161728-2.nat']
[WARNING: 2018-06-14 16:25:29 : native_msg] Number of HRV columns from header and derived from data are not consistent!
[WARNING: 2018-06-14 16:25:29 : native_msg] Number of columns read from header = 11136
[WARNING: 2018-06-14 16:25:29 : native_msg] Number of columns calculated from data = 5568
[DEBUG: 2018-06-14 16:25:29 : satpy.composites] Looking for composites config file seviri.yaml
[DEBUG: 2018-06-14 16:25:29 : satpy.composites] Looking for composites config file visir.yaml
Traceback (most recent call last):
  File "test_dsid.py", line 11, in <module>
    scn.load(dsid)
  File "/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/scene.py", line 692, in load
    level=level)
  File "/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/node.py", line 465, in find_dependencies
    n, unknowns = self._find_dependencies(key, **dfilter)
  File "/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/node.py", line 428, in _find_dependencies
    node = self._find_reader_dataset(dataset_key, **dfilter)
  File "/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/node.py", line 296, in _find_reader_dataset
    ds_id = reader_instance.get_dataset_key(dataset_key, **dfilter)
  File "/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/readers/yaml_reader.py", line 197, in get_dataset_key
    return get_key(key, self.ids.keys(), **kwargs)
  File "/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/readers/__init__.py", line 204, in get_key
    res = filter_keys_by_dataset_id(key, key_container)
  File "/opt/miniconda2/envs/satpy-env/lib/python3.6/site-packages/satpy-0.9.0a4.dev0-py3.6.egg/satpy/readers/__init__.py", line 133, in filter_keys_by_dataset_id
    if getattr(did, key) is not None:
AttributeError: 'NoneType' object has no attribute 'name'

[djhoese]

@marfel I talked with @mraspaud on slack and we figured out the issue he was having, but the way he suggested doing this is correct:

from glob import glob
dsid = DatasetID(name='VIS006', modifiers=('sunz_corrected',))
scn = Scene(filenames=glob("/home/a001673/data/satellite/Meteosat-11/seviri/lvl1.5/2018/02/28/HRIT/*"),
            reader='hrit_msg')
scn.load(dsid)

NOTE the command after `sunz_corrected' to make the modifiers argument a tuple.

[djhoese]

@marfel Could you show me the code that produced that error message? Specifically what is dsid?

[marfel]

from satpy import Scene, DatasetID
from glob import glob

basename = '/geo/MSG/offline/MSG3-SEV/2015/08/01/*2015080115*.nat'

dsid = DatasetID(name='VIS006', modifiers=('sunz_corrected',))
files = glob(basename)
for msgfile in files:
    scn = Scene(reader="native_msg", filenames=[msgfile])
    scn.load(dsid)

[djhoese]

@mraspaud Are you able to produce anything like this with the data files you have?

[marfel]

If you wish I can upload the *.nat file somewhere, so you can test the reader.

[mraspaud]

@sjoro might be better equiped, but I'll try with what I have

[mraspaud]

@marfel it should be scn.load([dsid]), not scn.load(dsid) :)

[marfel]

haha... that's a classic m-) Now it works.
I think I can live with that, thank you guys!