Add CREFL rayleigh corrector for ABI data #317

wroberts4 · 2018-06-04T14:54:12Z

I added the rayleigh correction to also work for abi data in order to incorporate a wider range of satpy functionality. The code I wrote was re-purposed from work done by Ralph Kuehn at SSEC.

Note: Ralph's code uses a different algorithm than that which is used by satpy. A research paper covering the algorithm that Ralph's code uses can be found here: https://www.atmos-meas-tech.net/6/2989/2013/

A couple of things to keep in mind:

Added modifiers in abi/viirs.yaml may need be re-purposed/removed in the future. Same goes for composites in abi.yaml.
The last 5 elements of bH2O in crefl_utils.py are actually aO2 values (in Ralph's code). Currently aO2 is not used in the satpy code since it would require restructuring of multiple features, so they are tagged at the end of bH2O.

Tests added
Tests passed
Passes git diff origin/master **/*py | flake8 --diff
Fully documented

stickler-ci · 2018-06-04T14:55:26Z

satpy/composites/crefl_utils.py

    aH2O = np.array(
        [0.000406601, 0.0015933, 0, 1.78644e-05, 0.00296457, 0.000617252,
-         0.000996563, 0.00222253, 0.00094005, 0.000563288, 0, 0, 0, 0, 0, 0])
+         0.000996563, 0.00222253, 0.00094005, 0.000563288, 0, 0, 0, 0, 0, 0,
+         2.4111e-003, 7.8454e-003*rg_fudge,7.9258e-3, 9.3392e-003, 2.53e-2])


E231 missing whitespace after ','

stickler-ci · 2018-06-04T14:55:27Z

satpy/composites/crefl_utils.py

+                     0.78812, 0.791204, 0.900564, 0.942907, 0, 0, 0, 0, 0, 0,
+                     # These are actually aO2 values for abi calculations
+                     1.2360e-003, 3.7296e-003, 177.7161e-006, 10.4899e-003, 1.63e-2])
+    # /*const float aO3[Nbands]={ 0.0711,    0.00313, 0.0104,     0.0930,   0, 0, 0, 0.00244, 0.00383, 0.0225, 0.0663, 0.0836, 0.0485, 0.0395, 0.0119, 0.00263};*/


E501 line too long (162 > 120 characters)

stickler-ci · 2018-06-04T14:55:27Z

satpy/composites/crefl_utils.py

-    #/*const float taur0[Nbands] = { 0.0507,  0.0164,  0.1915,  0.0948,  0.0036,  0.0012,  0.0004,  0.3109, 0.2375, 0.1596, 0.1131, 0.0994, 0.0446, 0.0416, 0.0286, 0.0155};*/
+                    0, 0, 0.0663, 0.0836, 0.0485, 0.0395, 0.0119, 0.00263,
+                    4.2869e-003, 25.6509e-003*rg_fudge, 802.4319e-006, 0.0000e+000, 2e-5])
+    # /*const float taur0[Nbands] = { 0.0507,  0.0164,  0.1915,  0.0948,  0.0036,  0.0012,  0.0004,  0.3109, 0.2375, 0.1596, 0.1131, 0.0994, 0.0446, 0.0416, 0.0286, 0.0155};*/


E501 line too long (175 > 120 characters)

stickler-ci · 2018-06-04T14:55:27Z

satpy/composites/crefl_utils.py

+def get_atm_variables_abi(mus, muv, phi, height, coeffs, G_O3, G_H2O, G_O2):
+    # coeffs = (ah2o, aO2, ao3, tau)
+    (ah2o, ao2, ao3, tau) = coeffs
+    missing_value = -999.0


F841 local variable 'missing_value' is assigned to but never used

stickler-ci · 2018-06-04T14:55:27Z

satpy/composites/crefl_utils.py

+    xfd = 0.958725775
+    xbeta2 = 0.5
+    as0 = [0.33243832, 0.16285370, -0.30924818, -0.10324388, 0.11493334,
+        -6.777104e-02, 1.577425e-03, -1.240906e-02, 3.241678e-02, -3.503695e-02]


E128 continuation line under-indented for visual indent

stickler-ci · 2018-06-04T14:55:28Z

satpy/composites/crefl_utils.py

+    fs2 = as2[0] + xlntaur * as2[1]
+    del xlntaur
+    trdown = np.exp(-taur / mus)
+    trup= np.exp(-taur / muv)


E225 missing whitespace around operator

stickler-ci · 2018-06-04T14:55:28Z

satpy/composites/crefl_utils.py

+    Ttotrayu = ((2 / 3. + muv) + (2 / 3. - muv) * trup) / (4 / 3. + taur)
+    Ttotrayd = ((2 / 3. + mus) + (2 / 3. - mus) * trdown) / (4 / 3. + taur)
+    if ao3 != 0:
+        tO3 =  np.exp(G_O3 * ao3)


E222 multiple spaces after operator

stickler-ci · 2018-06-04T14:55:29Z

satpy/composites/crefl_utils.py

+    if ah2o != 0:
+        tH2O = np.exp(G_H2O * ah2o)
+
+    tO2 =  np.exp(G_O2 * ao2)


E222 multiple spaces after operator

stickler-ci · 2018-06-04T14:55:29Z

satpy/composites/viirs.py

+            sensor_aa, sensor_za, solar_aa, solar_za = self.get_angles(vis)
+        else:
+            vis, sensor_aa, sensor_za, solar_aa, solar_za = self.check_areas(
+            datasets + optional_datasets)


E122 continuation line missing indentation or outdented

stickler-ci · 2018-06-04T14:55:29Z

satpy/composites/viirs.py

+            solar_aa = solar_aa.data
+            solar_za = solar_za.data
+        # angles must be xarrays
+        sensor_aa = xr.DataArray(sensor_aa, dims=['y','x'])


E231 missing whitespace after ','

codecov · 2018-06-04T15:11:48Z

Codecov Report

Merging #317 into master will increase coverage by 1.53%.
The diff coverage is 80%.

@@            Coverage Diff            @@
##           master    #317      +/-   ##
=========================================
+ Coverage   69.27%   70.8%   +1.53%     
=========================================
  Files         123     125       +2     
  Lines       15833   16270     +437     
=========================================
+ Hits        10968   11520     +552     
+ Misses       4865    4750     -115

Impacted Files	Coverage Δ
satpy/tests/test_crefl_utils.py	`0% <0%> (ø)`
satpy/enhancements/__init__.py	`77.18% <100%> (+2.18%)`	⬆️
satpy/tests/test_enhancements.py	`98.55% <100%> (+0.21%)`	⬆️
satpy/composites/crefl_utils.py	`83.62% <77.94%> (+83.62%)`	⬆️
satpy/readers/hdfeos_l1b.py	`14.71% <9.67%> (-0.7%)`	⬇️
satpy/composites/viirs.py	`84.09% <90%> (+5.39%)`	⬆️
satpy/tests/compositor_tests/test_viirs.py	`99.28% <99.32%> (+0.04%)`	⬆️
satpy/tests/test_writers.py	`97.68% <0%> (-0.03%)`	⬇️
satpy/tests/reader_tests/test_generic_image.py	`97.76% <0%> (ø)`
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 78314dc...cf5640b. Read the comment docs.

coveralls · 2018-06-04T15:14:31Z

Coverage increased (+1.5%) to 70.805% when pulling cf5640b on wroberts4:feature_abi_rayleigh into 78314dc on pytroll:master.

stickler-ci · 2018-06-04T15:24:22Z

satpy/composites/crefl_utils.py

+    xph3 = xfd * xbeta2 * 0.375 * (1.0 - mus * mus) * (1.0 - muv * muv)
+
+    fs01 = as0[0] + (mus + muv)*as0[1] + (mus * muv)*as0[2] + (mus * mus + muv * muv)*as0[3] + \
+           (mus * mus * muv * muv)*as0[4]


E127 continuation line over-indented for visual indent

stickler-ci · 2018-06-04T15:24:23Z

satpy/composites/crefl_utils.py

+    fs01 = as0[0] + (mus + muv)*as0[1] + (mus * muv)*as0[2] + (mus * mus + muv * muv)*as0[3] + \
+           (mus * mus * muv * muv)*as0[4]
+    fs02 = as0[5] + (mus + muv)*as0[6] + (mus * muv)*as0[7] + (mus * mus + muv * muv)*as0[8] + \
+           (mus * mus * muv * muv)*as0[9]


E127 continuation line over-indented for visual indent

djhoese · 2018-06-04T16:48:57Z

If I recall correctly @wroberts4 this still produces different results that Ralph's original code. Maybe you could try running the same data through this satpy code and Ralph's old code and show us screenshots of the results? Or I guess screenshots once they are equivalent.

Then just unittests to get the coverage score higher and maybe @adybbroe and @mraspaud would agree that this should maybe be moved to composites/__init__.py? Or maybe make the full jump to adding it to pyspectral?

mraspaud · 2018-06-08T06:55:34Z

About moving to composites/__init__.py I don't know, I was just looking at the file the other day, and thought it was starting to get big. Should we maybe start splitting it instead ?

djhoese · 2018-06-08T15:07:40Z

@mraspaud Sounds good to me. Wondering if we should have _rayleigh.py or maybe a atmospheric_correctors.py or something like that?

mraspaud · 2018-06-08T16:27:43Z

atmospheric_correctors.py sounds good

djhoese · 2018-06-11T16:23:09Z

satpy/composites/viirs.py

@@ -38,6 +39,14 @@
 LOG = logging.getLogger(__name__)


+class IncompatibleAreas(Exception):


This should have been imported from composites/__init__.py

stickler-ci · 2018-06-11T17:45:16Z

satpy/composites/crefl_utils.py

+    xph3 = xfd * xbeta2 * 0.375 * (1.0 - mus * mus) * (1.0 - muv * muv)
+
+    fs01 = as0[0] + (mus + muv)*as0[1] + (mus * muv)*as0[2] + (mus * mus + muv * muv)*as0[3] + \
+           (mus * mus * muv * muv)*as0[4]


E127 continuation line over-indented for visual indent

stickler-ci · 2018-06-12T18:28:40Z

satpy/composites/viirs.py

 import xarray.ufuncs as xu

 from satpy.composites import CompositeBase, GenericCompositor
 from satpy.config import get_environ_ancpath
 from satpy.dataset import combine_metadata

 LOG = logging.getLogger(__name__)
+from satpy.composites import IncompatibleAreas


E402 module level import not at top of file
F401 'satpy.composites.IncompatibleAreas' imported but unused

djhoese · 2018-06-19T02:00:31Z

satpy/composites/crefl_utils.py

+            # FIXME: if/when dask can support lazy index arrays then remove this
+            return sphalb0[index_arr]
+        sphalb_delayed = dask.delayed(_sphalb_index)(sphalb0, (taur / TAUSTEP4SPHALB + 0.5).astype(np.int32))
+        sphalb = da.from_delayed(sphalb_delayed, taur.shape, dtype=sphalb0.dtype)


I thought about this while working on a separate feature. I think you could get this to work with map_blocks if you provide index_arr as the first argument, specify the chunks=taur.chunks, and most importantly before passing the sphalb0 array do sphalb0 = sphalb0.rechunk(sphalb0.shape). We can talk about this in person later.

stickler-ci · 2018-06-22T19:51:06Z

satpy/tests/compositor_tests/test_viirs.py

@@ -224,6 +224,214 @@ def test_hncc_dnb(self):
            unique, [3.484797e-04, 9.507845e-03, 4.500016e+03])


+    def test_ReflectanceCorrector_abi(self):


E303 too many blank lines (2)

stickler-ci · 2018-06-22T19:51:06Z

satpy/tests/compositor_tests/test_viirs.py

+        from pyresample.geometry import AreaDefinition
+        from satpy.composites.viirs import ReflectanceCorrector
+        from satpy import DatasetID
+        ref_cor = ReflectanceCorrector(dem_filename= '_fake.hdf', optional_prerequisites= [


E251 unexpected spaces around keyword / parameter equals

stickler-ci · 2018-06-22T19:51:06Z

satpy/tests/compositor_tests/test_viirs.py

+            DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=None, polarization=None, calibration=None,
+                      level=None, modifiers=None),
+            DatasetID(name='solar_zenith_angle', wavelength=None, resolution=None, polarization=None, calibration=None,
+                      level=None, modifiers=None)], name= 'C01', prerequisites= [],


E251 unexpected spaces around keyword / parameter equals

stickler-ci · 2018-06-22T19:51:06Z

satpy/tests/compositor_tests/test_viirs.py

+                      level=None, modifiers=None),
+            DatasetID(name='solar_zenith_angle', wavelength=None, resolution=None, polarization=None, calibration=None,
+                      level=None, modifiers=None)], name= 'C01', prerequisites= [],
+             wavelength= (0.45, 0.47, 0.49), resolution= 1000, calibration= 'reflectance',


E251 unexpected spaces around keyword / parameter equals

stickler-ci · 2018-06-22T19:51:06Z

satpy/tests/compositor_tests/test_viirs.py

+            DatasetID(name='solar_zenith_angle', wavelength=None, resolution=None, polarization=None, calibration=None,
+                      level=None, modifiers=None)], name= 'C01', prerequisites= [],
+             wavelength= (0.45, 0.47, 0.49), resolution= 1000, calibration= 'reflectance',
+                                       modifiers= ('sunz_corrected', 'rayleigh_corrected_crefl',), sensor= 'abi')


E131 continuation line unaligned for hanging indent
E251 unexpected spaces around keyword / parameter equals

stickler-ci · 2018-06-22T19:51:08Z

satpy/tests/compositor_tests/test_viirs.py

+                                'polarization':         None,
+                                'sensor':               'viirs',
+                                'units':                units,
+                                'start_time':           datetime.datetime(2012,2,25,18,1,24,570942),


E231 missing whitespace after ','

stickler-ci · 2018-06-22T19:51:08Z

satpy/tests/compositor_tests/test_viirs.py

+                                'sensor':               'viirs',
+                                'units':                units,
+                                'start_time':           datetime.datetime(2012,2,25,18,1,24,570942),
+                                'end_time':             datetime.datetime(2012,2,25,18,11,21,175760),


E231 missing whitespace after ','

stickler-ci · 2018-06-22T19:51:08Z

satpy/tests/compositor_tests/test_viirs.py

+                          'solar_azimuth_angle', 'solar_azimuth_angle', 'degrees')
+        c05 = make_xarray(None, (), None, ['gitco', 'gimgo'], 'All_Data/{file_group}_All/SolarZenithAngle',
+                          'solar_zenith_angle', 'solar_zenith_angle', 'degrees')
+        res = ref_cor([c01], [c02,c03,c04,c05])


E231 missing whitespace after ','

stickler-ci · 2018-06-22T19:51:09Z

satpy/tests/compositor_tests/test_viirs.py

+        self.assertEqual(res.attrs['platform_name'], 'Suomi-NPP')
+        self.assertEqual(res.attrs['sensor'], 'viirs')
+        self.assertEqual(res.attrs['units'], '%')
+        self.assertEqual(res.attrs['start_time'], datetime.datetime(2012,2,25,18,1,24,570942))


E231 missing whitespace after ','

stickler-ci · 2018-06-22T19:51:09Z

satpy/tests/compositor_tests/test_viirs.py

+        self.assertEqual(res.attrs['sensor'], 'viirs')
+        self.assertEqual(res.attrs['units'], '%')
+        self.assertEqual(res.attrs['start_time'], datetime.datetime(2012,2,25,18,1,24,570942))
+        self.assertEqual(res.attrs['end_time'], datetime.datetime(2012,2,25,18,11,21,175760))


E231 missing whitespace after ','

stickler-ci · 2018-06-22T20:09:03Z

satpy/tests/compositor_tests/test_viirs.py

+            DatasetID(name='solar_zenith_angle', wavelength=None, resolution=None, polarization=None, calibration=None,
+                      level=None, modifiers=None)], name='C01', prerequisites=[],
+             wavelength=(0.45, 0.47, 0.49), resolution=1000, calibration='reflectance',
+                      modifiers=('sunz_corrected', 'rayleigh_corrected_crefl',), sensor='abi')


E131 continuation line unaligned for hanging indent

stickler-ci · 2018-06-22T20:09:03Z

satpy/tests/compositor_tests/test_viirs.py

+        from satpy.composites.viirs import ReflectanceCorrector
+        from satpy import DatasetID
+        ref_cor = ReflectanceCorrector(dem_filename='_fake.hdf', optional_prerequisites=[
+         DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),


E501 line too long (149 > 120 characters)

stickler-ci · 2018-06-22T20:09:03Z

satpy/tests/compositor_tests/test_viirs.py

+        from satpy import DatasetID
+        ref_cor = ReflectanceCorrector(dem_filename='_fake.hdf', optional_prerequisites=[
+         DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),


E501 line too long (148 > 120 characters)

stickler-ci · 2018-06-22T20:09:03Z

satpy/tests/compositor_tests/test_viirs.py

+        ref_cor = ReflectanceCorrector(dem_filename='_fake.hdf', optional_prerequisites=[
+         DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),


E501 line too long (145 > 120 characters)

stickler-ci · 2018-06-22T20:09:03Z

satpy/tests/compositor_tests/test_viirs.py

+         DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None)],


E501 line too long (145 > 120 characters)

stickler-ci · 2018-06-22T20:09:03Z

satpy/tests/compositor_tests/test_viirs.py

+         DatasetID(name='satellite_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None)],
+        name='I01', prerequisites=[], wavelength=(0.6, 0.64, 0.68), resolution=371, calibration='reflectance',


E122 continuation line missing indentation or outdented

stickler-ci · 2018-06-22T20:09:03Z

satpy/tests/compositor_tests/test_viirs.py

+         DatasetID(name='solar_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None),
+         DatasetID(name='solar_zenith_angle', wavelength=None, resolution=371, polarization=None, calibration=None, level=None, modifiers=None)],
+        name='I01', prerequisites=[], wavelength=(0.6, 0.64, 0.68), resolution=371, calibration='reflectance',
+        modifiers=('sunz_corrected_iband', 'rayleigh_corrected_crefl_iband'), sensor='viirs')


E122 continuation line missing indentation or outdented

stickler-ci · 2018-07-18T16:10:29Z

satpy/tests/compositor_tests/test_viirs.py

+        def make_xarray(wavelength, modifiers, resolution, file_type, name, calibration):
+            return xr.DataArray(dnb,
+                                dims=('y', 'x'),
+                                attrs={ 'wavelength': wavelength, 'level': None, 'modifiers': modifiers,


E201 whitespace after '{'

djhoese

Looks pretty good. I have a few things I'd like you to change and other things that I was just pointing out as reminders to myself to make issues for.

djhoese · 2018-07-26T17:30:26Z

satpy/composites/crefl_utils.py

-            # otherwise make it the same size as the other arrays
-            sphalb = np.repeat(np.repeat(sphalb, factor, axis=0), factor, axis=1)
+        from satpy.composites import IncompatibleAreas
+        raise IncompatibleAreas("")


I would say remove this entire if block. This should theoretically not be possible when called from the modifier (because of check_areas).

djhoese · 2018-07-26T17:33:55Z

satpy/composites/viirs.py

-            avg_elevation = nc.variables[self.dem_sds][:]
+            # average elevation is stored as a 16-bit signed integer but with
+            # scale factor 1 and offset 0, convert it to float here
+            avg_elevation = nc.variables[self.dem_sds][:].astype(np.float)


I think in the long run we'll want this to be dask arrays, but for now let's not worry about it. It probably isn't a performance issue anyway.

djhoese · 2018-07-26T17:39:14Z

satpy/enhancements/__init__.py

@@ -110,6 +110,23 @@ def apply_enhancement(data, func, exclude=None, separate=False,
    return data


+# pointed to by generic.yaml
+def crefl_scaling(img, **kwargs):


As discussed on slack I think we'll have to make another issue to combine this with the existing crude stretch so that it allows for multiple "control" points.

djhoese · 2018-07-26T17:40:14Z

satpy/etc/composites/abi.yaml

@@ -1,6 +1,15 @@
 sensor_name: visir/abi

 modifiers:
+  # TODO: fix


Remove # TODO. Also future note, usually a todo of fixing something is just # FIXME: description 😉

djhoese · 2018-07-26T17:44:12Z

satpy/etc/enhancements/generic.yaml

@@ -198,7 +207,7 @@ enhancements:
    operations:
    - name: stretch
      method: *stretchfun
-      kwargs: {stretch: linear}
+      kwargs: {stretch: crude, min_stretch: 0, max_stretch: 120}


@mraspaud @adybbroe @pnuu Thoughts on changing the default reflectance scaling to a static 0 to 120?

djhoese · 2018-07-26T17:45:11Z

satpy/node.py

@@ -423,12 +423,10 @@ def _find_dependencies(self, dataset_key, **dfilter):
        except KeyError:
            # exact dataset isn't loaded, let's load it below
            pass
-


Why were these lines removed? Automatic by some auto-correcting tool? Please add the blank lines back in unless they were style errors.

djhoese · 2018-07-26T17:46:15Z

satpy/tests/test_crefl_utils.py

+import unittest
+
+
+class testCreflUtils(unittest.TestCase):


Since your my student @wroberts4 I'm going to ask you to fix the name of this class and the name of your test functions to be PEP8 compatible. TestCreflUtils and test_get_atm_variables_abi

Try to review other tests that you've added.

stickler-ci · 2018-07-27T13:42:43Z

satpy/node.py

@@ -472,5 +472,5 @@ def find_dependencies(self, dataset_keys, **dfilter):
                continue

            self.add_child(self, n)
-
+            


W293 blank line contains whitespace

stickler-ci · 2018-07-27T13:42:43Z

satpy/tests/test_crefl_utils.py

+
+    loader = unittest.TestLoader()
+    mysuite = unittest.TestSuite()
+    mysuite.addTest(loader.loadTestsFromTestCase(testCreflUtils))


F821 undefined name 'testCreflUtils'

djhoese

Things left to decide:

Should the default reflectance enhancement be changed? If not, then this has to be fixed before this is merged.
The bug fix in resample.py should probably be included in a bug fix release of 0.9.x (cherry-picked if possible).

@wroberts4

First fixed by @wroberts4 in pytroll#317 but was not cherry-pick-able

mraspaud

just a comment

mraspaud · 2018-08-17T13:25:30Z

satpy/composites/crefl_utils.py

@@ -331,6 +363,73 @@ def get_atm_variables(mus, muv, phi, height, coeffs):
    return sphalb, rhoray, TtotraytH2O, tOG


+def get_atm_variables_abi(mus, muv, phi, height, coeffs, G_O3, G_H2O, G_O2):


there seems to be a lot of duplicated code in this function from existing function, can we factorise ?

You are correct! I cooked something up and will push it later today or Monday (after I test to make sure I didn't break it).

stickler-ci · 2018-08-17T20:02:42Z

satpy/composites/crefl_utils.py

+        G_H2O = G_calc(solar_zenith, a_H2O) + G_calc(sensor_zenith, a_H2O)
+        G_O2 = G_calc(solar_zenith, a_O2) + G_calc(sensor_zenith, a_O2)
+        # Note: bh2o values are actually ao2 values for abi
+        sphalb, rhoray, TtotraytH2O, tOG = get_atm_variables_abi(mus, muv, phi, height, *coeffs, G_O3, G_H2O, G_O2)


E999 SyntaxError: only named arguments may follow *expression

djhoese · 2018-08-19T18:55:46Z

satpy/etc/readers/hdfeos_l1b.yaml

-    calibration: reflectance
+    calibration:
+      reflectance:
+        units: "%"


Do we know that the units coming out of this are % (0-~120) and not 1 (0 - ~1.2)?

djhoese · 2018-08-21T20:55:32Z

satpy/etc/composites/modis.yaml

@@ -12,15 +21,15 @@ composites:
      modifiers: [sunz_corrected]
    standard_name: true_color

-  true_color:
+  true_color_crefl:


Please reset this to be true_color and to use rayleigh_corrected instead of rayleigh_corrected_crefl. Feel free to leave the rayleigh_corrected_crefl definition above. Make sure that this true color composite still loads successfully.

djhoese · 2018-08-23T01:25:09Z

satpy/tests/test_crefl_utils.py

+                                                                 0.0043149700000000004, 0.0037296,
+                                                                 0.014107995000000002, 0.052349)
+        if abs(np.array(sphalb) - 0.045213532544630494) >= 1e-10:
+            raise AssertionError('{} is not within {} of {}'.format(np.array(sphalb), 1e-10, 0.045213532544630494))


Sorry, I should have noticed these earlier. These should be replaced with assert statements or self.assertX calls.

djhoese · 2018-08-23T01:26:52Z

satpy/tests/compositor_tests/test_viirs.py

+        self.assertEqual(ref_cor.attrs['sensor'], 'abi')
+        self.assertEqual(ref_cor.attrs['prerequisites'], [])
+        self.assertEqual(ref_cor.attrs['optional_prerequisites'], [
+            DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=None, polarization=None,


The default for DatasetID is None so these can all be left empty (except for maybe modifiers which defaults to an empty tuple).

djhoese · 2018-08-23T01:27:42Z

satpy/tests/compositor_tests/test_viirs.py

+        self.assertEqual(res.attrs['ancillary_variables'], [])
+        data = res.values
+        if abs(np.mean(data) - 29.907390988422513) >= 1e-10:
+            raise AssertionError('{} is not within {} of {}'.format(np.mean(data), 1e-10, 29.907390988422513))


Replace this with self.assertX call or regular assert statement at least.

djhoese · 2018-08-23T01:28:10Z

satpy/tests/compositor_tests/test_viirs.py

+        from satpy.composites.viirs import ReflectanceCorrector
+        from satpy import DatasetID
+        ref_cor = ReflectanceCorrector(dem_filename='_fake.hdf', optional_prerequisites=[
+         DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None, calibration=None,


No need to list each None element here.

djhoese · 2018-08-23T01:28:19Z

satpy/tests/compositor_tests/test_viirs.py

+        self.assertEqual(ref_cor.attrs['prerequisites'], [])
+        self.assertEqual(ref_cor.attrs['optional_prerequisites'], [
+            DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=371, polarization=None,
+                      calibration=None, level=None, modifiers=None),


djhoese · 2018-08-23T01:28:57Z

satpy/tests/compositor_tests/test_viirs.py

+
+        area, dnb = self.data_area_ref_corrector()
+
+        def make_xarray(wavelength, modifiers, calibration, file_type, file_key, name, standard_name, units):


Maybe it is github's weird formatting but this looks like it needs a self as the first argument.

djhoese · 2018-08-23T01:29:23Z

satpy/tests/compositor_tests/test_viirs.py

+        self.assertEqual(res.attrs['ancillary_variables'], [])
+        data = res.values
+        if abs(np.mean(data) - 40.7578684169142) >= 1e-10:
+            raise AssertionError('{} is not within {} of {}'.format(np.mean(data), 1e-10, 40.7578684169142))


Replace with self.assertX

djhoese · 2018-08-23T01:29:34Z

satpy/tests/compositor_tests/test_viirs.py

+        import datetime
+        from satpy.composites.viirs import ReflectanceCorrector
+        from satpy import DatasetID
+        sataa_did = DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=None, polarization=None,


None parameters can be removed

djhoese · 2018-08-23T01:29:43Z

satpy/tests/compositor_tests/test_viirs.py

+        self.assertEqual(ref_cor.attrs['sensor'], 'modis')
+        self.assertEqual(ref_cor.attrs['prerequisites'], [])
+        self.assertEqual(ref_cor.attrs['optional_prerequisites'], [
+            DatasetID(name='satellite_azimuth_angle', wavelength=None, resolution=None, polarization=None,


Add rayleigh correction to abi data

240799f

stickler-ci reviewed Jun 4, 2018

View reviewed changes

Fix small issues raised by Stickler

5e0f3d0

Split lines that are greater than 120 characters

ca8040c

stickler-ci reviewed Jun 4, 2018

View reviewed changes

mraspaud assigned wroberts4 Jun 5, 2018

mraspaud added enhancement code enhancements, features, improvements component:compositors labels Jun 5, 2018

Fix crefl_utils.py bugs

046d159

djhoese reviewed Jun 11, 2018

View reviewed changes

Fix crefl image and add a crefl visual enhancement

eafdea3

stickler-ci reviewed Jun 11, 2018

View reviewed changes

Fix crefl_utils.py bugs and improve performance

143118f

stickler-ci reviewed Jun 12, 2018

View reviewed changes

djhoese reviewed Jun 19, 2018

View reviewed changes

wroberts4 added 2 commits June 22, 2018 13:59

Merge remote-tracking branch 'origin/master' into feature_abi_rayleigh

75ef307

Add tests for crefl enhancement and ReflectanceCorrector

d571ffd

stickler-ci reviewed Jun 22, 2018

View reviewed changes

Fix Stickler syntax complaints

72b04f0

stickler-ci reviewed Jun 22, 2018

View reviewed changes

wroberts4 added 4 commits June 22, 2018 15:16

Fix Stickler complaints round two

e9d12b6

Fix final Stickler complains for test_viirs

41344a0

Fix Stickler complains for comma separation

9383c78

Add tests, fix minor bugs, and improve crefl enhancement

0588df5

Fix Stickler complaints

9f88c06

stickler-ci reviewed Jul 18, 2018

View reviewed changes

wroberts4 added 2 commits July 18, 2018 11:21

Fix Stickler complaints

4b3f0d1

Fix pull request test fails

3c53b28

djhoese added this to the v0.10 milestone Jul 26, 2018

djhoese requested changes Jul 26, 2018

View reviewed changes

Fix naming, spacing, and redundant code

0558ccf

stickler-ci reviewed Jul 27, 2018

View reviewed changes

wroberts4 added 2 commits July 27, 2018 08:50

Fix Stickler complaints

34de4b0

Fix large solar zenith angles and use p2g scaling for crefl enhancement

7290dde

djhoese approved these changes Jul 31, 2018

View reviewed changes

djhoese changed the title ~~[WIP] Add Rayleigh Correction to abi Data~~ Add CREFL rayleigh corrector for ABI data Jul 31, 2018

Revert reflectance back to linear stretch

a1bf104

djhoese added a commit to djhoese/satpy that referenced this pull request Aug 14, 2018

Fix native resampler when a dataset doesn't have 'y' and 'x' coords

b56fbb3

First fixed by @wroberts4 in pytroll#317 but was not cherry-pick-able

djhoese mentioned this pull request Aug 14, 2018

Fix native resampler using SwathDefinition as an AreaDefinition #391

Merged

3 tasks

mraspaud reviewed Aug 17, 2018

View reviewed changes

wroberts4 added 3 commits August 17, 2018 14:43

Make crefl_utils.py less redundant

b3d29c6

Make crefl_utils.py less redundant

fc4fbbd

Make crefl_utils.py less redundant

f6c3fa1

stickler-ci reviewed Aug 17, 2018

View reviewed changes

Fix Stickler complaints

e43b976

djhoese reviewed Aug 19, 2018

View reviewed changes

Merge branch 'master' into feature_abi_rayleigh

e4ee9ac

djhoese requested changes Aug 21, 2018

View reviewed changes

djhoese added 2 commits August 22, 2018 20:09

Cleanup modis true color composite

f533748

Update HDFEOS L1B reader to have proper units and calibrations

d081554

djhoese requested changes Aug 23, 2018

View reviewed changes

Clean-up tests

cf5640b

djhoese approved these changes Aug 27, 2018

View reviewed changes

djhoese merged commit fa72f94 into pytroll:master Aug 27, 2018

		@@ -38,6 +39,14 @@
		LOG = logging.getLogger(__name__)


		class IncompatibleAreas(Exception):

		@@ -224,6 +224,214 @@ def test_hncc_dnb(self):
		unique, [3.484797e-04, 9.507845e-03, 4.500016e+03])


		def test_ReflectanceCorrector_abi(self):

		@@ -472,5 +472,5 @@ def find_dependencies(self, dataset_keys, **dfilter):
		continue

		self.add_child(self, n)

		@@ -331,6 +363,73 @@ def get_atm_variables(mus, muv, phi, height, coeffs):
		return sphalb, rhoray, TtotraytH2O, tOG


		def get_atm_variables_abi(mus, muv, phi, height, coeffs, G_O3, G_H2O, G_O2):


		area, dnb = self.data_area_ref_corrector()

		def make_xarray(wavelength, modifiers, calibration, file_type, file_key, name, standard_name, units):

Add CREFL rayleigh corrector for ABI data #317

Add CREFL rayleigh corrector for ABI data #317

Conversation

wroberts4 commented Jun 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 4, 2018 • edited Loading

Codecov Report

coveralls commented Jun 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djhoese commented Jun 4, 2018

mraspaud commented Jun 8, 2018

djhoese commented Jun 8, 2018

mraspaud commented Jun 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djhoese left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djhoese left a comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wroberts4 commented Jun 4, 2018 •

edited

Loading

codecov bot commented Jun 4, 2018 •

edited

Loading

coveralls commented Jun 4, 2018 •

edited

Loading