Add xarray/dask support to DayNightCompositor #217
Conversation
satpy/composites/__init__.py
Outdated
| except IndexError: | ||
| from pyorbital.astronomy import cos_zen | ||
| LOG.debug("Computing sun zenith angles.") | ||
| lons, lats = day_data.attrs["area"].get_lonlats_dask(CHUNK_SIZE) |
There was a problem hiding this comment.
You should probably pass day_data.chunks as the chunk size for potentially better performance (assuming day_data is 2D).
There was a problem hiding this comment.
I tried that, but day_data is usually, but not always, a 3D dataset, so the chunking doesn't necessarily match. I'll take a closer look when I'm back at office.
There was a problem hiding this comment.
Oh ok. Makes sense. You might be able to get away with day_data.sel(bands=day_data['bands'][0]).chunks...maybe.
There was a problem hiding this comment.
Put another way, what if you got one 2D array from the 3D array and use the chunks from that. Dask might give you logical chunks. The point is if you can pass the actual chunks then calculations should be overall faster since the lons/lats will be at the same chunk size as the data.
satpy/resample.py
Outdated
| @@ -673,7 +675,7 @@ def mask_source_lonlats(source_def, mask): | |||
| # the data may have additional masked pixels | |||
| # let's compare them to see if we can use the same area | |||
| # assume lons and lats mask are the same | |||
| if mask and isinstance(source_geo_def, SwathDefinition): | |||
| if mask is not None and isinstance(source_geo_def, SwathDefinition): | |||
There was a problem hiding this comment.
I think @mraspaud merged his fix for this. You may want to rebase and take his changes which are slightly different than this.
There was a problem hiding this comment.
Hmm, I thought I did a pull on develop, but maybe I was still on this feature branch. I'll rebase before I continue with the comments you left.
| alpha = new_data[0].copy() | ||
| alpha.data = da.ones((data.sizes['y'], | ||
| data.sizes['x']), | ||
| chunks=new_data[0].chunks) |
There was a problem hiding this comment.
I know you had questions about this. This is probably one of the easier ways to do this. You could have also taken all the dask arrays for each band, added an alpha with da.ones(...) and then did da.stack to concatenate them. You would then have to create a new xarray and pass all of the dims, attrs, and coords from the original data. At least that's the only way I know how to do it.
There was a problem hiding this comment.
Well, at least it works! I'll try if I can wrap my head around what you described. Sounds complicated.
satpy/composites/__init__.py
Outdated
| new_data['bands'] = ['R', 'G', 'B'] | ||
| data = new_data | ||
| # Add alpha band | ||
| if 'A' not in data.bands.data and 'A' in bands.data: |
There was a problem hiding this comment.
Could you user data['bands'] instead ?
satpy/composites/__init__.py
Outdated
| data = new_data | ||
| # Add alpha band | ||
| if 'A' not in data.bands.data and 'A' in bands.data: | ||
| new_data = [data.sel(bands=band) for band in data.bands.data] |
satpy/resample.py
Outdated
| @@ -673,7 +675,7 @@ def mask_source_lonlats(source_def, mask): | |||
| # the data may have additional masked pixels | |||
| # let's compare them to see if we can use the same area | |||
| # assume lons and lats mask are the same | |||
| if mask and isinstance(source_geo_def, SwathDefinition): | |||
| if mask is not None and isinstance(source_geo_def, SwathDefinition): | |||
…into feature-xarray-DNC
mraspaud
added
enhancement
This PR adds xarray and dask support to DayNightCompositor.
git diff origin/develop **/*py | flake8 --diff