Creating a Scene object with NOAA-15/18 data

[Krangaas]

Code Sample, a minimal, complete, and verifiable piece of code

import os
from satpy import Scene
from satpy.utils import debug_on
debug_on()

BASEDIR = "C:\scrypts\K5"
FILES = os.path.join(BASEDIR, "hrpt_noaa15_20180330_1618_03429.l1b")

global_scene = Scene(reader="avhrr_aapp_l1b", filenames = [FILES])

Problem description

I'm having difficulty creating a scene object with NOAA-15 data, I'm unsure of what the problem is. I get the same error when using NOAA-18 data.

Expected Output

I want to create a scene object, load the channels and create a composite of the data.

Actual Result, Traceback if applicable

(base) C:\scrypts>python SatPy_NOAA.py
[DEBUG: 2018-06-29 14:18:38 : satpy.scene] Setting 'PPP_CONFIG_DIR' to 'C:\Users\magnus\Anaconda2\lib\site-packages\satpy\etc'
[DEBUG: 2018-06-29 14:18:38 : satpy.readers] Reading ['C:\Users\magnus\Anaconda2\lib\site-packages\satpy\etc\readers\avhrr_aapp_l1b.yaml', 'C:\Users\magnus\Anaconda2\lib\site-packages\satpy\etc\readers\avhrr_aapp_l1b.yaml', 'C:\Users\ma
gnus\Anaconda2\lib\site-packages\satpy\etc\readers\avhrr_aapp_l1b.yaml']
[DEBUG: 2018-06-29 14:18:38 : satpy.readers.yaml_reader] Assigning to avhrr_aapp_l1b: ['C:\scrypts\K5\hrpt_noaa15_20180330_1618_03429.l1b']
Traceback (most recent call last):
File "SatPy_NOAA.py", line 10, in
global_scene = Scene(reader="avhrr_aapp_l1b", filenames = [FILES])
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\scene.py", line 150, in init
reader_kwargs=reader_kwargs)
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\scene.py", line 195, in create_reader_instances
ppp_config_dir=self.ppp_config_dir)
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\readers_init_.py", line 589, in load_readers
reader_instance.create_filehandlers(loadables)
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\readers\yaml_reader.py", line 487, in create_filehandlers
filename_set)
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\readers\yaml_reader.py", line 474, in new_filehandlers_for_filetype
return list(filtered_iter)
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\readers\yaml_reader.py", line 447, in filter_fh_by_metadata
for filehandler in filehandlers:
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\readers\yaml_reader.py", line 391, in new_filehandler_instances
yield filetype_cls(filename, filename_info, filetype_info, *req_fh)
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\readers\aapp_l1b.py", line 78, in init
self.read()
File "C:\Users\magnus\Anaconda2\lib\site-packages\satpy\readers\aapp_l1b.py", line 135, in read
data = np.memmap(fp_, dtype=_SCANTYPE, offset=22016, mode="r")
File "C:\Users\magnus\Anaconda2\lib\site-packages\numpy\core\memmap.py", line 236, in new
raise ValueError("Size of available data is not a "
ValueError: Size of available data is not a multiple of the data-type size.

Versions of Python, package at hand and relevant dependencies

OS: Windows 7
SatPy Version: 0.9.0b0
PyResample Version: 1.10.0
Python version: 2.7

[mraspaud]

Thanks for reporting this. Would you please confirm that these datafiles was created with AAPP? if so, could share one of the files for further testing?

[Krangaas]

Unfortunately I won't be able to confirm this until Monday when my colleague becomes available, sorry for the inconvenience.

[mraspaud]

no problem, Monday is fine

[Krangaas]

Great! I'll get back to you then.

[Krangaas]

@mraspaud I can confirm that this datafile was created with AAPP, would you still like me to share the file with you?

[mraspaud]

yes please!

[Krangaas]

Ok, I'll send it to you via mail.

[mraspaud]

Thanks, I got it. Looking at the file with od -a | less I can see that the file doesn't seem to comply with the aapp data format described here:
https://www.nwpsaf.eu/site/software/aapp/documentation/ (data formats)
Indeed, the header of the file is supposed to start with three characters indicating the processing station, followed by a blank.
you got this:

0000000   N   O   T   A   P   P   N   O   T   A   P   P   P   P   2   0
0000020   0   1  sp   3   0   D   2   0   0   1   0   9   2   0   N   S
0000040   S   .   H   R   P   T   .   N   D   .   D   0   1   2   5   7
0000060   .   S   0   2   5   5   .   E   0   3   0   9   .   B   5   0
0000100   2   4   3   4   3   .   W   I  sp  sp   T   A   L   L   A   L
0000120   L   A   L   L  sp   A   L   L  sp   A   L   L  sp   A   L   L
0000140   Y   Y   Y   Y   Y   Y   Y   Y   Y   Y   Y   Y   Y   Y   Y   Y
0000160   Y   Y   Y   Y   Y   1   6  sp  sp  sp  sp  sp  sp  sp  sp  sp
0000200  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp
0000220  sp  sp   B   +   9   0   -   9   0   -   1   8   0   +   1   8
0000240   0   A   A   P   P  sp   N   O   A   A  sp   l   e   v   e   l
0000260   1   b  sp  sp  sp  sp   2   2   0   1   6  sp  sp   2   7   9
0000300   9  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp
0000320  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp  sp
*
0001000   S   P   T  sp etx nul   S bel   $ nul nul   V nul nul soh nul
0001020 nul nul nul nul nul nul   S   P   T   .   H   R   P   T   .   N
0001040   K   .   D   1   8   0   8   9   .   S   1   6   1   8   .   E
0001060   1   6   2   6   .   B   0   3   4   2   9   2   9   .   W   E
0001100   B   0   3   4   2   9   2   9 eot nul nul nul etx nul eot nul
0001120   ]   a nul nul   b bel   Y nul   !   p del etx   ]   a nul nul
0001140   b bel   Y nul   e  so bel etx  si nul   l nul nul nul nul nul
0001160 nul nul nul nul nul   w nul nul nul nul nul nul nul nul nul nul

But you can see that what we are interested in starts at byte 512 (0001000)... So something is adding a chunk of 512 bytes in the beginning of your files it seems.

Removing those 512 bytes from the file makes it readable by satpy:

[Krangaas]

Thanks again! perhaps the same header is being added to the NOAA-18 data aswell, I'll have a look at it when I get the chance.

[Krangaas]

@mraspaud I'm have some difficulty implementing the code for removing the 512 first bytes, seeing as how you managed to plot the satellite data you were able to remove or skip the header?

[mraspaud]

@Krangaas I used bash to do that, with tail -c... But is it practical for you to process all the files this way ? or do you need to do it on the fly ? Maybe with a check to see if the file starts with NOTAPP, like this in aapp_l1b.py:

    def read(self):
        """Read the data.
        """
        tic = datetime.now()
        offset = 0
        with open(self.filename, "rb") as fp_:
            if fp_.read(6) == b"NOTAPP":
                offset = 512
        header = np.memmap(self.filename, dtype=_HEADERTYPE, offset=offset, mode="r", shape=(1, ))
        data = np.memmap(self.filename, dtype=_SCANTYPE, offset=offset + 22016, mode="r")
        logger.debug("Reading time %s", str(datetime.now() - tic))

        self._header = header
        self._data = data

But to accept that into satpy, we would have to make sure to know the source of these bytes and that they are always 512 bytes long...

[Krangaas]

I'm not sure why the 512 byte chunk is added to the data but I believe it will be avoided in the future. I'll try what you suggested, thanks!