Fails to build on Android, Pixel 3, Termux

[filbranden]

Hi!

I tried to install pyzmq (actually pulled as a dependency of jupyterlab) in my Android phone, a Pixel 3, running the Termux Linux environment/terminal emulator.

I tried to install it using:

$ python3 -m pip install --user pyzmq

The failures are to build the C/C++ code under the bundled/ tree.

First, this failed due to the strlcpy() symbol being redefined. Android ships a libc called "bionic" which actually derives from BSD libc, so it does include its own strlcpy().

I downloaded it locally and tried to hack around that error, then I found the next around the tipc code (don't remember exactly if it was something like tipc_zone or something like that.)

I hacked through that one and the third problem was around pthread_setaffinity_np().

After these three, I got it to build, but then it didn't seem to work... Running jupyter lab would start spewing errors and they looked related to zmq.

These are the characteristics of the system:

$ uname -a
Linux localhost 4.9.232-gdc3a39a45881-ab6891296 #0 SMP PREEMPT Thu Oct 8 04:02:27 UTC aarch64 Android
$ python3 -V
Python 3.8.5
$ gcc -v
clang version 10.0.1 (https://github.com/termux/termux-packages 2f1431d824f449f09ef50aa365cdd7ed5b8c159d)
Target: aarch-unknown-linux-android
Thread model: posix
InstalledDir: /data/data/com.termux/files/usr/bin

I noticed there are directories under buildutils/include_* (including one for linux-armv). If you'd like me to try to configure my system and send you an appropriate platform.hpp to bundle into pyzmq I'd be happy to help with that.

---

I actually found a workaround... There's a package for libzmq in Termux, so I just installed that, with:

$ pkg install libzmq

After that, pip install --user pyzmq worked just fine (I'm assuming it detected the lib was already present) and jupyter lab (which uses pyzmq) worked fine as well.

So perhaps just stopping the install and issuing an instruction to install the binary library would be an appropriate course of action here? Not sure how this can be reliably detected... But breaking with a "Please install xyz" is definitely better than with a puzzling compilation error...

Cheers!
Filipe

[daviscyd]

@filbranden I had the same issue...the strlcpy error, pthread_setaffinity_np error, and the tipc_addr and tipc_cluster errors...with libzmq installed. Did you configure anything in the default termux environment that allowed the install to work?

[filbranden]

@daviscyd No, not that I'm aware of... I simply used the pkg install libzmq and after that python3 -m pip install --user pyzmq worked for me, right away.

I don't know if there was anything left on my environment, since I ended up downloading the tarball and doing some local pip operations... I don't think I did left anything, but maybe I did? I guess I could try that on a second phone to see if I can reproduce it... I happen to have one lying around that needs a factory reset, so excellent one to try and see what happens...

In any case, I'm happy to try to help fix this however I can. I don't know that I understood exactly how the bundled helpers work... But if sending a platform.hpp for this platform would help, I'd be happy to try to provide one (given instructions on how to correctly generate it.)

[filbranden]

@daviscyd I just tested it on a fresh phone and it worked... Phone was Pixel 2 and didn't have any of it installed.

Started by installing Termux. Then inside it, I ran:

$ pkg install python

Then:

$ python3 -m pip install --user pyzmq

Which failed with the strlcpy error:

    bundled/zeromq/src/compat.hpp:44:1: error: static declaration of 'strlcpy' follows non-static declaration
    ...
    /data/data/com.termux/files/usr/include/string.h:146:8: note: previous declaration is here
    size_t strlcpy(char* __dst, const char* __src, size_t __n);
    1 error generated.
    error: command '/data/data/com.termux/files/usr/bin/aarch64-linux-android-clang++' failed with exit code 1

Then I installed the library from the binary package:

$ pkg install libzmq

And finally repeated:

$ python3 -m pip install --user pyzmq

Whch worked fine, with:

$ python3 -m pip install --user pyzmq
Collecting pyzmq
  Using cached pyzmq-20.0.0.tar.gz (1.2 MB)
Using legacy 'setup.py install' for pyzmq, since package 'wheel' is not installed.
Installing collected packages: pyzmq
    Running setup.py install for pyzmq ... done
Successfully installed pyzmq-20.0.0

And finally:

$ python3
Python 3.9.1 (default, Dec  9 2020, 13:22:01)
[Clang 9.0.8 (https://android.googlesource.com/toolchain/llvm-project 98c855489 on linux
>>> import zmq
>>> zmq.zmq_version()
'4.3.3'
>>>

[daviscyd]

@filbranden Thanks for testing this. I just uninstalled python and re-installed and I'm still getting the same error...with libzmq installed.

I'll uninstall Termux and re-try again, following the steps that you used. If possible, could you run printenv on your Pixel and post the output?

[filbranden]

@daviscyd Maybe try pkg update too... It's possible you have a libzmq version older than 4.3.3 and pyzmq looks for that one?

The one I have shows up as:

$ pkg show libzmq
Package: libzmq
Version: 4.3.3
Maintainer: Termux members @termux
Installed-Size: 1028 kB
Depends: libc++, libsodium
Breaks: libzmq-dev
Replaces: libzmq-dev
Homepage: https://zeromq.org/
Download-Size: 316 kB
APT-Manual-Installed: yes
APT-Sources: https://dl.bintray.com/termux/termux-packages-24 stable/main aarch64 Packages
Description: Fast messaging system built on sockets. C and C++ bindings. aka 0MQ, ZMQ.

In any case, it would be nice to fix the bundled one to work on Android + Termux...

[daviscyd]

@filbranden . I ran pkg upgrade, then pkg update before starting the build. I followed the steps you posted but the build still fails at the same error:

bundled/zeromq/src/compat.hpp:44:1: error: static declaration of 'strlcpy' follows non-static declaration
    ...
    /data/data/com.termux/files/usr/include/string.h:146:8: note: previous declaration is here
    size_t strlcpy(char* __dst, const char* __src, size_t __n);
    1 error generated.

My guess is that, after you install libzmq and run python3 -m pip install --user pyzmq, the build picks up the newly installed libzmq.
Which makes me wonder why my build doesn't pick up the newly installed libzmq.

Also, while I have the same version of libzmq installed (4.3.3) it's from this repo:
APT-Sources: https://termux.org/packages stable/main arm Packages

[filbranden]

@daviscyd Looking at setup.py, looks like it uses pkg-config to locate the library.

Maybe try pkg install pkg-config ?

[daviscyd]

@filbranden pkg-config is already installed on my device. I'll take a look at setup.py, maybe there's a missing variable.

[filbranden]

@daviscyd Your package source (https://termux.org/packages stable/main arm Packages) suggests it's using "arm" architecture, which means 32-bit ARM.

If your uname -a output shows aarch64, then you have a 64-bit ARM chip.

If that's the case, it's possible that the setup.py logic thinks it's cross-compiling (libraries don't match host architecture) and that will trigger a build from bundle. Maybe this explains it?

[daviscyd]

@filbranden Good catch. The CPU architecture is ARM64 (armv8l) but uname -a output is Linux localhost 4.14.87+ #1 SMP PREEMPT Sat Dec 12 17:46:56 UTC 2020 armv8l Android

So the question is: where's the logic that determines if bundled zmq is used or not?

When you run pkg-config --exists libzmq what is the result?

UPDATE: Specifying --install-option="--zmq=<path> as per the build output also fails.

[daviscyd]

I downloaded the pyzmq module and used setup.py to build locally and found the error preventing the module install from using the installed lzmq.

arm-linux-androideabi-clang build/temp.linux-armv8l-3.9/scratch/data/data/com.termux/files/usr/tmp/timer_createlz92f8lw.o -o build/temp.linux-armv8l-3.9/scratch/a.out 

arm-linux-androideabi-clang -mfloat-abi=softfp -mfpu=vfpv3-d16 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -march=armv7-a -mfpu=neon -mfloat-abi=softfp -mthumb -fstack-protector-strong -O3 -march=armv7-a -mfpu=neon -mfloat-abi=softfp -mthumb -fstack-protector-strong -O3 -fPIE -fPIC -fPIC -I/data/data/com.termux/files/usr/include -Izmq/utils -c build/temp.linux-armv8l-3.9/scratch/vers.c -o build/temp.linux-armv8l-3.9/scratch/vers.o arm-linux-androideabi-clang build/temp.linux-armv8l-3.9/scratch/vers.o -L/data/data/com.termux/files/usr/lib -R/data/data/com.termux/files/usr/lib -lzmq -o build/temp.linux-armv8l-3.9/scratch/vers

Error running version detection script:  "build/temp.linux-armv8l-3.9/scratch/vers": error: Android 5.0 and later only support position-independent executables (-fPIE).

I'm not sure if the above is an issue with the compiler options, the build script or the installed libzmq

[bkt92]

I also have same issue when install pyzmq:

| ^~~~~~~~~ In file included from /data/data/com.termux/files/usr/include/python3.9/Python.h:151, from zmq/devices/monitoredqueue.c:24:
/data/data/com.termux/files/usr/include/python3.9/abstract.h:343:17: note: declared here
343 | PyAPI_FUNC(int) PyObject_AsReadBuffer(PyObject *obj,
| ^~~~~~~~~~~~~~~~~~~~~
error: command '/data/data/com.termux/files/usr/bin/arm-linux-androideabi-clang++' failed with exit code 1

[daviscyd]

UPDATE: Downloading sources and building/installing libzmq on-device allowed pip install pyzmq to run successfully

[sacrrie]

UPDATE: Downloading sources and building/installing libzmq on-device allowed pip install pyzmq to run successfully

I tried to manually build from source of libzmq on-device using termux-chroot but still didn't solve my problem. Can you shed a little bit more light on how installed it?

[daviscyd]

I tried to manually build from source of libzmq on-device using termux-chroot but still didn't solve my problem. Can you shed a little bit more light on how installed it?

I don't use termux-chroot and didn't use it during the build. I just downloaded the libzmq sources, ran configure and make install

[sacrrie]

I tried to manually build from source of libzmq on-device using termux-chroot but still didn't solve my problem. Can you shed a little bit more light on how installed it?

I don't use termux-chroot and didn't use it during the build. I just downloaded the libzmq sources, ran configure and make install

Thank you for your reply. When I just configure and make install without termux-chroot, I got following errors:
/data/data/com.termux/files/usr/bin/mkdir: cannot create directory ‘/usr’: Read-only file system make[2]: *** [Makefile:3695: install-libLTLIBRARIES] Error 1 make[2]: Leaving directory '/data/data/com.termux/files/home/libzmq' make[1]: *** [Makefile:9641: install-am] Error 2 make[1]: Leaving directory '/data/data/com.termux/files/home/libzmq' make: *** [Makefile:8216: install-recursive] Error 1
Could you please tell me how you managed to install it?

[yubrajbhoi]

@sacrrie I was having this same problem and compiling with clang instead of arm-linux-androideabi-clang fixed it for me. My exact command was CC=clang python setup.py build and CC=clang pip install pyzmq should also work. You don't need to compile libzmq from source.

termux/termux-packages#6241 (comment)

[PallavRastogi]

I am also getting strlcpy error, I have libzmq 4.3.4 installed , still I cannot install pyzmq.
First it showed can't build wheel error, then I did something and now it shows strlcpy error

[PallavRastogi]

Architecture- arm32
And I am installing on termux

[anzhi0708]

@sacrrie I was having this same problem and compiling with clang instead of arm-linux-androideabi-clang fixed it for me. My exact command was CC=clang python setup.py build and CC=clang pip install pyzmq should also work. You don't need to compile libzmq from source.

termux/termux-packages#6241 (comment)

Solved my problem. Thanks!

pkg install git
git clone https://github.com/zeromq/pyzmq.git
cd pyzmq/
CC=clang python setup.py build
CC=clang pip install pyzmq

[mercer00]

Thanks! , It worked!

[futurk]

it worked, thank you for the contribution :)

[anfe67]

@filbranden thank you very much, I was in the same situation, and your post saved my day.

[minrk]

Thanks for everyone's reporting here! It doesn't sound like there's anything to do on this repo, so I'm goign to close, but feel free to continue to post what worked for you here, and open a new issue if you think there's something reasonable that pyzmq can do to help.