Problem with installing Pyproj / proj

[gersmit]

Dear sir or madam,
My name is Ger Smit from the Netherlands.
I have problem with pyproj / proj.
I use Raspberry Pi, Python, for making screne from data with satpy.
No problems til now.
I install satpy, try to run python script and get error.
I I was in contact with @djhoese about this and he referred me to you
ERROR: Minimum supported proj version is 6.2.0, installed version is 4.9.3. For more information see: https://pyproj4.github.io/pyproj/stable/installation.htm

Then when i run the script i get error :

Traceback (most recent call last):
File "/home/pi/a-satpy-testen/cursus_seviri.py", line 83, in
scn.load(scn.all_dataset_names()[1:]) #HEEL belangrijk
File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1163, in load
self._read_datasets_from_storage(**kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1183, in _read_datasets_from_storage
return self._read_dataset_nodes_from_storage(nodes, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1189, in _read_dataset_nodes_from_storage
loaded_datasets = self._load_datasets_by_readers(reader_datasets, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1214, in _load_datasets_by_readers
new_datasets = reader_instance.load(ds_ids, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 945, in load
ds = self._load_dataset_with_area(dsid, coords, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 1109, in _load_dataset_with_area
ds = super(GEOFlippableFileYAMLReader, self)._load_dataset_with_area(dsid, coords, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area
ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data
proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 689, in _load_dataset
projectable = fh.get_dataset(dsid, ds_info)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_l1b_native.py", line 458, in get_dataset
self._update_attrs(dataset, dataset_info)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_l1b_native.py", line 582, in _update_attrs
actual_lon, actual_lat, actual_alt = self.satpos
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_l1b_native.py", line 610, in satpos
semi_minor_axis=self.mda['projection_parameters']['b']
File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_base.py", line 703, in get_satpos
proj='geocent', a=semi_major_axis, b=semi_minor_axis, units='m'
File "/home/pi/.local/lib/python3.7/site-packages/pyproj/crs.py", line 436, in init
super(CRS, self).init(projstring)
File "pyproj/_crs.pyx", line 1738, in pyproj._crs._CRS.init
pyproj.exceptions.CRSError: Invalid projection: +proj=geocent +a=6378169.0 +b=6356583.800000001 +units=m +type=crs: (Internal Proj Error: proj_create: Invalid b value)

Please can you explaine to me how to install or solved pyproj / proj in combination with satpy.
I try a lot withoud result.

Thanks in advange
@gersmit

[jdkloe]

Hi Ger, thanks for your report. It seems to me you are struggling with different incompatible package versions. The proj version you have installed is too old and cannot be used with a recent pyproj module. So the choices you have is to install a newer proj version or to install an older pyproj version.
In addition you mention satpy, but you don't mention its version or how you obtained it (is it part of raspbian/debian or did you use pip or conda or ...). This is another thing to be considerd here.
Without this information it is hard to help you.

[jdkloe]

Looking at https://github.com/pytroll/satpy/blob/main/setup.py I see that satpy requires 'pyproj>=2.2'
On https://pyproj4.github.io/pyproj/stable/history.html you can see that this pyproj version has a minimum PROJ version of 6.1.0.
So a recent satpy module cannot be used together with proj version is 4.9.3 as you have installed now.
My recommendation would be to first try to update the proj library, or re-install it using a (much) newer version.

[snowman2]

There are some instructions for building PROJ from source on Raspbian here: piwheels/packages#60. Also, feel free to comment on that issue if you would like to see wheels.

[snowman2]

Also, the version compatibility matrix is likely a helpful resource: https://pyproj4.github.io/pyproj/stable/installation.html#installing-from-source

[gersmit]

Thanks for the quick responses, it all sounds good and strange that I haven't had any problems in previous situations.

I made a new sdcard for the Raspberry Pi with a x64 Buster image.

Now how can I best proceed to install proj, pyproj and satpy. I can install Satpy with pip or conda but also see that RPI has it as software as "Python Package for earth-observing satellite data processing Python3-satpy-0.11.1-2".
I've never used this one.

I hope you can help me, possible is there an easy way (wget or something) to install proj/pyproj, my RPI program knowledge is there but should not go too deep.

The attachement is what RPI also has inside as PROJ

software. I dont't no if i can use that.

[djhoese]

@gersmit These all look like very old versions of these software libraries including Satpy. I think you should install Python (3.7 or greater) and then use traditional Python tools like pip (or conda is you install python that way). The nice thing about using conda (conda-forge) is that all of the binary dependencies are built for you (like the PROJ C library).

[gersmit]

@djhoese , I already had a little bit of that. Didn't know Raspbian was carrying this either.

I'm using Python 3.7 but don't really know how to install proj.
I find this : https://qiita.com/XPT60/items/be4e3d9f9f58d17cf968. Translated from Japanese, the explanation is understandable to me, but is this what I should have in combination with Satpy?

I will install conda tomorrow and again follow the way you described. If it works for someone else, it should work for me too....yes, right

[snowman2]

@gersmit you can try downloading, unzipping and installing this wheel: pyproj-2.4.1-cp37-cp37m-linux_armv7l.whl.zip

[gersmit]

I make install like @snowman2 explain, pyproj 2.4.1
I update & upgrade etc
I install Satpy 0.29.0 with pip3
Get error:
Traceback (most recent call last):
File "/home/pi/a-satpy-testen/cursus2_seviri.py", line 105, in
from satpy.scene import Scene
File "/home/pi/.local/lib/python3.7/site-packages/satpy/init.py", line 26, in
from satpy.dataset import DataID, DataQuery # noqa
File "/home/pi/.local/lib/python3.7/site-packages/satpy/dataset/init.py", line 22, in
from .metadata import combine_metadata # noqa
File "/home/pi/.local/lib/python3.7/site-packages/satpy/dataset/metadata.py", line 26, in
from satpy.writers.utils import flatten_dict
File "/home/pi/.local/lib/python3.7/site-packages/satpy/writers/init.py", line 29, in
import xarray as xr
File "/home/pi/.local/lib/python3.7/site-packages/xarray/init.py", line 3, in
from . import testing, tutorial, ufuncs
File "/home/pi/.local/lib/python3.7/site-packages/xarray/testing.py", line 8, in
from xarray.core import duck_array_ops, formatting, utils
File "/home/pi/.local/lib/python3.7/site-packages/xarray/core/duck_array_ops.py", line 15, in
from . import dask_array_compat, dask_array_ops, dtypes, npcompat, nputils
File "/home/pi/.local/lib/python3.7/site-packages/xarray/core/npcompat.py", line 39, in
from numpy.typing import ArrayLike, DTypeLike
File "/usr/local/lib/python3.7/dist-packages/numpy/typing/init.py", line 316, in
from ._dtype_like import (
File "/usr/local/lib/python3.7/dist-packages/numpy/typing/_dtype_like.py", line 95, in
class _SupportsDType(Generic[_DType_co]):
NameError: name '_DType_co' is not defined

What is happening now and can i solved this?
This is the example from

[djhoese]

Looks like numpy isn't installed correctly.

[snowman2]

I wonder if this will help: piwheels/packages#60 (comment)

[gersmit]

Made new image RPI Buster x32. Install fresh pip install satpy and get error during install
scm_config
use_scm_version={
File "/tmp/easy_install-O_reoh/setuptools_scm-6.0.1/src/setuptools_scm/init.py", line 8, in
File "/tmp/easy_install-O_reoh/setuptools_scm-6.0.1/src/setuptools_scm/config.py", line 6, in
File "/tmp/easy_install-O_reoh/setuptools_scm-6.0.1/src/setuptools_scm/utils.py", line 41
print(*k)
^
SyntaxError: invalid syntax

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-EPnzAQ/zarr/

I don't no anymore what i have to do first

[snowman2]

Seems like a python version issue.

I recommend making a virtual environment:

python3 -m venv venv
. venv/bin/activate
python -m pip install -U pip
python -m pip install scipy

[gersmit]

@snowman2 sorry but no result, install stops on the same row.
The img on the RPI is totally clean
Is it maybe beter to install first the Satpy prerequisites, and what are these?

I installed the same almost 1 year ago on RPI4-4GB, no problems
That is working til now Version 0.23.0 (2020/09/18), pyproj 1.9.6, proj 0.1.0.

Now i try it on a clean Buster x64 and clean Buster x32 on RPI 3b with the problems

[snowman2]

Are you able to generate a requirements.txt using pip freeze from the old environment? That might help you re-create one in a new environment.

[gersmit]

Step by step i am going.
Later i explane what i have done.
The reader is trying to start but get error--> missing FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3/dist-packages/pyspectral/etc/pyspectral.yaml'
Is it possible to install this?

The error complete = :
MSG4-SEVI-MSG15-0100-NA-20210807142743.311000000Z-NA.nat
{'seviri_l1b_native': ['MSG4-SEVI-MSG15-0100-NA-20210807142743.311000000Z-NA.nat']}
Traceback (most recent call last):
File "/home/pi/a-satpy-testen/cursus_seviri.py", line 76, in
scn = Scene(filenames=files)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 114, in init
comps, mods = self._composite_loader.load_compositors(self.attrs['sensor'])
File "/home/pi/.local/lib/python3.7/site-packages/satpy/composites/config_loader.py", line 249, in load_compositors
self.load_sensor_composites(sensor_name)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/composites/config_loader.py", line 204, in load_sensor_composites
self._load_config(composite_configs)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/composites/config_loader.py", line 274, in load_config
conf = recursive_dict_update(conf, yaml.load(conf_file, Loader=UnsafeLoader))
File "/home/pi/.local/lib/python3.7/site-packages/yaml/init.py", line 114, in load
return loader.get_single_data()
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 51, in get_single_data
return self.construct_document(node)
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 60, in construct_document
for dummy in generator:
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
value = self.construct_mapping(node)
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 218, in construct_mapping
return super().construct_mapping(node, deep=deep)
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 143, in construct_mapping
value = self.construct_object(value_node, deep=deep)
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 102, in construct_object
data = constructor(self, tag_suffix, node)
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 570, in construct_python_name
return self.find_python_name(suffix, node.start_mark)
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 719, in find_python_name
return super(UnsafeConstructor, self).find_python_name(name, mark, unsafe=True)
File "/home/pi/.local/lib/python3.7/site-packages/yaml/constructor.py", line 551, in find_python_name
import(module_name)
File "/home/pi/.local/lib/python3.7/site-packages/satpy/modifiers/init.py", line 23, in
from .spectral import NIREmissivePartFromReflectance, NIRReflectance # noqa: F401
File "/home/pi/.local/lib/python3.7/site-packages/satpy/modifiers/spectral.py", line 27, in
from pyspectral.near_infrared_reflectance import Calculator
File "/usr/lib/python3/dist-packages/pyspectral/near_infrared_reflectance.py", line 33, in
from pyspectral.utils import BANDNAMES, get_bandname_from_wavelength
File "/usr/lib/python3/dist-packages/pyspectral/utils.py", line 174, in
CONF = get_config()
File "/usr/lib/python3/dist-packages/pyspectral/config.py", line 74, in get_config
with open(configfile, 'r') as fp:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3/dist-packages/pyspectral/etc/pyspectral.yaml'

[djhoese]

This is a config file that comes with the pyspectral package. It should be included when you install the package. How did you install satpy/pyspectral?

[gersmit]

pip3 install pyspectral

[djhoese]

I notice that your traceback bounces between python packages in /home/pi/.local and then looks at pyspectral in /usr/lib/python3. Would it be possible for you to make sure you install all new packages in /home/pi/.local? You may be using an old version of pyspectral or are installing it at the system level when you have all the other libraries installed in your user directory.

pip3 install pyspectral should work, but since it didn't let's start with putting it in /home/pi/.local like you did the other packages. You should be able to do that by doing pip3 install --user pyspectral.

[snowman2]

#893 (comment)

☝️

[gersmit]

Question:
On the working RPI i find '/usr/lib/python3/site-packages/pyspectral/etc/pyspectral.yaml'
Why is asking the new RPI '/usr/lib/python3/dist-packages/pyspectral/etc/pyspectral.yaml'

[djhoese]

Not sure. It mind depend on what version of python or how python was installed on the system level. Does /usr/lib/python3/dist-packages/ exist on the new RPI? Either way, you should try following @snowman2's suggestion that he linked to above about creating a virtual environment.

[gersmit]

I use suggestie @snowman2 ----> (venv) pi@raspberrypi:~ $
/usr/lib/python3/dist-packages/ exist on the new RPI
ERROR stay:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3/dist-packages/pyspectral/etc/pyspectral.yaml'
DIR 'pyspectral 'has no dir 'etc'

[djhoese]

Did you do the pip3 install before or after activating the venv?

pip3 install pyspectral should have put the pyspectral package in your venv, not in the system /usr/lib directory. 😕

[gersmit]

small question:
It is starting to run and have to ugrade pyresample

ERROR: Command errored out with exit status 1:
command: /usr/bin/python3 /home/pi/.local/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp2ltorcw9
cwd: /tmp/pip-install-s8je6qez/pyproj_457c6b5ae5fc4ba997045117a2ba9858
Complete output (1 lines):
ERROR: Minimum supported proj version is 6.2.0, installed version is 4.9.3. For more information see: https://pyproj4.github.io/pyproj/stable/installation.html

I try to install proj 6.2.0
Then

Building with CMake

cd proj
mkdir build
cd build
cmake ..
cmake --build .

pi@raspberrypi:~/proj/build $ cmake --build .
make: *** Geen doelen opgegeven en geen Makefile gevonden. Gestopt.

How do i use the last row cmake --build . . I don't know how to deal with this

[snowman2]

@gersmit, with the wheel #893 (comment), you don't need to install PROJ, correct?

[gersmit]

@snowman2 , new img dd 13-8-2021, 18:00 hr, RPI buster X64
pi@raspberrypi:~/Downloads $ cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 44.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

processor : 1
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 44.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

processor : 2
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 44.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

processor : 3
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 44.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

Hardware : BCM2835
Revision : a02082
Serial : 00000000a0bb581a
Model : Raspberry Pi 3 Model B Rev 1.2
pi@raspberrypi:/Downloads $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@raspberrypi:/Downloads $ cat /sys/firmware/devicetree/base/model
Raspberry Pi 3 Model B Rev 1.2pi@raspberrypi:~/Downloads $

pi@raspberrypi:~/Downloads $ sudo pip install pyproj-2.4.1-cp37-cp37m-linux_armv7l.whl
pyproj-2.4.1-cp37-cp37m-linux_armv7l.whl is not a supported wheel on this platfor

install satpy
Installing build dependencies ... done
Complete output from command python setup.py egg_info:
Proj executable not found. Please set PROJ_DIR variable.

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-CbGDI3/pyproj/

it can't and shouldn't be that hard. What the hell am I doing wrong. Whereas before there were no problems at all.
I can still try with Stretch. This will be difficult for the amateur. help me

stops on

[snowman2]

Does this work for you? piwheels/packages#60 (comment)

[gersmit]

@snowman2
get ssh error

pi@raspberrypi:~ $ git clone [email protected]:pyproj4/pyproj.git
Cloning into 'pyproj'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

[snowman2]

Maybe https might work better for you:

git clone https://github.com/pyproj4/pyproj.git

[gersmit]

Hi @snowman2 ,
What have i to do to install the pyproj.git to make it running?

[snowman2]

Base on: piwheels/packages#60 (comment)

I am thinking you will likely have more success if you do this:

mkdir -p ~/projects/sqlite
wget https://www.sqlite.org/2018/sqlite-autoconf-3240000.tar.gz
mv sqlite-autoconf-3240000.tar.gz projects/sqlite/
cd ~/projects/sqlite/
tar xf sqlite-autoconf-3240000.tar.gz 
cd sqlite-autoconf-3240000/
./configure
make
make install
cd $HOME
export PROJ_DIR=$HOME/projects/proj
mkdir -p $PROJ_DIR
wget https://github.com/pyproj4/pyproj/archive/refs/tags/v2.6.1rel.tar.gz
tar xf v2.6.1rel.tar.gz
cd pyproj-2.6.1rel/
bash ci/travis/proj-dl-and-compile 6.3.2
python setup.py install

[gersmit]

@snowman2, I installed as you explane.
At the end error -->
root@raspberrypi:~/pyproj-2.6.1rel# python setup.py install
Traceback (most recent call last):
File "setup.py", line 219, in
long_description=get_long_description(),
File "setup.py", line 210, in get_long_description
with open("README.md", encoding="utf-8") as ld_file:
TypeError: 'encoding' is an invalid keyword argument for this function

What now, do i have change order.

Then when i try to install Satpy again an Error
And i cannot see that a Python 3.7 dir is installed in dir /home/pi/.local/lib

ERROR: Command errored out with exit status 1:
command: /usr/bin/python /home/pi/.local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpGUVTul
cwd: /tmp/pip-install-WJY3vT/pyproj
Complete output (1 lines):
Proj executable not found. Please set PROJ_DIR variable.

ERROR: Command errored out with exit status 1: /usr/bin/python /home/pi/.local/lib/python2.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpGUVTul Check the logs for full command output.

Why is it difficult now to install and for a couple of months ago easy?

[snowman2]

#893 (comment)

You need python 3. A virtual environment is best ☝️

[snowman2]

I think this is your best bet for this to be easier in the future: piwheels/packages#60

[gersmit]

@snowman2 , Thanks for all the help.
I am busy now with installing.
After this i make a new image, can you tell me what is the best order of work to install Satpy good

[djhoese]

Installing pyproj (and maybe rasterio) should be the hardest parts of install Satpy hopefully. The rest of the dependencies are relatively simple compared to pyproj and rasterio. As long as you use a virtualenv with Python 3.7+ and follow @snowman2's instructions for installing pyproj then the rest of this shouldn't be too bad (pip install satpy). It seems like so far it has just been difficult to get your build to keep using Python 3.

[gersmit]

Small question:
is there a list for for example what is channel C2 --> IR_039 ? etcetera

[djhoese]

This is a Satpy specific question and if you have more they should maybe go on the satpy repository (so we can stop bugging Alan). As for the answer: It all depends on the sensor you are talking about. Are you saying C2 as in Channel 2 of the SEVIRI instrument or are you saying C02 as in the ABI Channel 2 band data provided by Satpy. You can typically google something like " channel list" and find some listing of an instrument's channels and their wavelengths. For example I found this for SEVIRI: https://www.eumetsat.int/0-degree-service

[snowman2]

Glad to hear that you have pyproj installed. Going to close this issue since it is resolved. I recommend asking satpy questions in their recommended channels.