[WIP] 64 bit iOS device support

[Sjors]

This should make it possible to build for iPhone 8, iPhone X and other iOs devices with that or new chips.

Fixes #11720

[Sjors]

It seems I need to make some additional change...

make HOST=aarch64-apple-darwin14 NO_QT=1
heartbeat_test.c => ../test/heartbeat_test.c
making links in engines...
making links in tools...
make[2]: Nothing to be done for `links'.
generating dummy tests (if needed)...
make[2]: Nothing to be done for `generate'.

Since you've disabled or enabled at least one algorithm, you need to do
the following before building:

	make depend

Configured for iphoneos-cross.
Building openssl...
making all in crypto...
/usr/bin/perl ../util/mkbuildinf.pl "-I. -I.. -I../include  -DOPENSSL_THREADS -D_REENTRANT -pipe -O2 -I/Volumes/SSD/src/bitcoin/depends/aarch64-apple-darwin14/include -O3 -isysroot /SDKs/ -fomit-frame-pointer -fno-common" "iphoneos-cross" >buildinf.h
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -mmacosx-version-min=10.8 -I. -I.. -I../include  -DOPENSSL_THREADS -D_REENTRANT -pipe -O2 -I/Volumes/SSD/src/bitcoin/depends/aarch64-apple-darwin14/include -O3 -isysroot /SDKs/ -fomit-frame-pointer -fno-common   -c -o cryptlib.o cryptlib.c
clang: warning: no such sysroot directory: '/SDKs/' [-Wmissing-sysroot]
In file included from cryptlib.c:117:
./cryptlib.h:62:10: fatal error: 'stdlib.h' file not found
#include <stdlib.h>
         ^~~~~~~~~~
1 error generated.
make[2]: *** [cryptlib.o] Error 1
make[1]: *** [build_crypto] Error 1
make: *** [/Volumes/SSD/src/bitcoin/depends/work/build/aarch64-apple-darwin14/openssl/1.0.1k-df65369e03c/./.stamp_built] Error 2

Full log.

[fanquake]

@Sjors Are you still working on this?

[Sjors]

@fanquake waiting for hints.

[laanwj]

@theuni can you take a look, looks like @Sjors ran into a brick wall here

[jonasschnelli]

I think this PR alone makes little sense. IMO including the openssl change in #11720 seems after solving a real use case.

[theuni]

See here for enough to get cross builds from Linux working: theuni@3907e52

I can confirm that I get valid aarch64/armv7 binaries using that. Though I have no idea if they work :)
Edit: That's building depends with NO_QT=1. I assume the Qt build is horribly broken.

Native builds are more painful, as for cross builds we can assume sdk paths, versions, compilers, etc. I'll save those for a next step.

[ken2812221]

How about change make command to

$(MAKE) -j1 depend build_libs libcrypto.pc libssl.pc openssl.pc

[DrahtBot]

The last travis run for this pull request was 144 days ago and is thus outdated. To trigger a fresh travis build, this pull request should be closed and re-opened.

[maflcko]

@Sjors Are you still working on this?

[Sjors]

Rebased and incorporated
theuni@3907e52 (some of it has already been merged). I left out armv7 because Apple dropped support for that in iOs 11. I renamed to PR to cover all depends, rather than just OpenSSL.

make will look in depends/SDKs by default, so you have to set SDK_PATH or add a symlink: ln -s /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs SDKs

Note that if you use SDK_PATH and then change it, that won't take effect until you do a make clean HOST=aarch64-apple-darwin14.

I'm ignoring UPNP for now, so the following command is what I'd like to see working:

make HOST=aarch64-apple-darwin14 NO_QT=1 NO_UPNP=1

Boost 1.64 seems to build 32 bit binaries, which iOs 11 doesn't like. I bumped it from 1.64 to 1.66, as suggested here. I can move that to a separate PR once everything works.

For OpenSSL, the following SED command leads to a "sed: RE error: illegal byte sequence" error on macOS; I probably need to adjust the syntax.

  sed -i.old "s/-isysroot.*(CROSS_SDK)//" Configure

BDB isn't happy, though I can avoid that by dropping wallet support (hopefully adding it in later).:

checking for mutexes... UNIX/fcntl
configure: WARNING: NO SHARED LATCH IMPLEMENTATION FOUND FOR THIS PLATFORM.
configure: error: Unable to find a mutex implementation
make: *** [/Users/bitcoin/dev/bitcoind/depends/work/build/aarch64-apple-darwin14/bdb/4.8.30-0f0d7f59f89/build_unix/.stamp_configured] Error 1

Once OpenSSL works, the next step would be:

./configure --prefix=`pwd`/depends/aarch64-apple-darwin14`

Testing this on a real device would require building the rest of the application, so I agree with @jonasschnelli that this PR ultimately needs to be combined with whatever solves #11720.

[Sjors]

As for cross-compilation from Ubuntu:

The Gitian build instructions use an old version of Xcode. Nowadays it seems you need to use the App Store to download it. To extract the iOs SDK on macOS machine:

tar -C /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/ -czf iPhoneOS.sdk.tar.gz iPhoneOS.sdk iPhoneOS11.4.sdk

Then copy it to the Ubuntu machine and extract into depends/SDKs.

make NO_QT=1 NO_UPNP=1 HOST=aarch64-apple-darwin14

Unfortunately this fails for me with:

    "clang++" "-target" "aarch64-apple-darwin14" "-arch" "arm64" "-miphoneos-version-min=11.4" "--sysroot" "/home/gitianuser/bitcoin/depends/SDKs/iPhoneOS11.4.sdk" "-mlinker-version=253.9" "-stdlib=libc++"   -std=c++11 -fvisibility=hidden     -I/home/gitianuser/bitcoin/depends/aarch64-apple-darwin14/include     -O3 -Wall -pedantic -gdwarf-2 -fexceptions -Wno-long-long -pthread -Wextra -Wno-long-long -Wno-unused-parameter -Wunused-function -fpermissive -pedantic -DBOOST_ALL_NO_LIB=1 -DBOOST_ATOMIC_STATIC_LINK=1 -DBOOST_SYSTEM_STATIC_LINK=1 -DBOOST_THREAD_BUILD_LIB=1 -DBOOST_THREAD_DONT_USE_CHRONO -DBOOST_THREAD_POSIX -DNDEBUG  -I"." -c -o "bin.v2/libs/thread/build/darwin-gnu-4.2.1/release/link-static/threadapi-pthread/threading-multi/pthread/once.o" "libs/thread/src/pthread/once.cpp"

clang: error: invalid version number in '-miphoneos-version-min=11.4'
...failed darwin.compile.c++ bin.v2/libs/thread/build/darwin-gnu-4.2.1/release/link-static/threadapi-pthread/threading-multi/pthread/once.o...

Maybe I did something wrong with copying the SDK? The directory depends/SDKs/iPhoneOS11.4.sdk contains Entitlements.plist Library SDKSettings.plist System usr.

[Sjors]

Prepending LC_ALL=C before the sed did the tick.

Next problem, when running make:

  CXX      libbitcoin_util_a-utiltime.o
util.cpp:1138:18: error: 'system' is unavailable: not available on iOS
    int nErr = ::system(strCommand.c_str());
                 ^
/Users/bitcoin/dev/bitcoind/depends/SDKs/iPhoneOS11.4.sdk/usr/include/stdlib.h:195:6: note: 'system' has been explicitly marked unavailable here
int      system(const char *) __DARWIN_ALIAS_C(system);

That's used for -alertnotify, -blocknotify and -walletnotify, which are probably fine to disable on iOs.

Hopefully we don't need too many other workarounds in the code to handle iOs.

I also need to figure out how to actually test this on a device. The simulator is also an issue, but can wait, since it needs x86 architecture while still using the iOs SDK.

[Sjors]

This also works with a symlink.

[Sjors]

Disabling those notify commands seems to be the only workaround needed, although it's hard to tell without running a real device. The latter is not that easy, because iOs doesn't allow launching arbitrary executables, which is the approach ABcore uses.

I do get a flood of the following warnings:

ld: warning: direct access in function 'boost::unit_test::utils::print_escaped_cdata(std::__1::basic_ostream<char, std::__1::char_traits<char> >&, boost::unit_test::basic_cstring<char const>)' 
from file '/Users/bitcoin/dev/bitcoind/depends/aarch64-apple-darwin14/lib/libboost_unit_test_framework-mt.a(xml_log_formatter.o)' to 
global weak symbol 'boost::unit_test::basic_cstring<char const>::null_str()::null' from file 'test/test_bitcoin-addrman_tests.o' means the weak symbol cannot be overridden at runtime. 
This was likely caused by different translation units being compiled with
different visibility settings.

[Sjors]

It's probably easiest to work with headers and compiled .a files: http://www.swiftprogrammer.info/swift_call_cpp.html#step1

Although that's beyond the scope of this project, I'd like to try to get this to work so that we know if this PR actually works. A minimum PoC would just print IBD progress in a console.

Given the following files:

• src/libbitcoin_cli.a
• src/libbitcoin_common.a
• src/libbitcoin_consensus.a
• src/libbitcoin_server.a
• src/libbitcoin_util.a

What do I use for the corresponding header files? And is libbitcoin_server.a the entry point for an RPC server running on the device?

[Sjors]

Rebased, but it's not compiling for me anymore:

Undefined symbols for architecture arm64:
  "_getauxval", referenced from:
      crc32c::Extend(unsigned int, unsigned char const*, unsigned long) in libcrc32c.a(libcrc32c_a-crc32c.o)
ld: symbol(s) not found for architecture arm64

[DrahtBot]

Needs rebase

[DrahtBot]

🐙 This pull request conflicts with the target branch and needs rebase.

[Sjors]

I'm declaring this up for grabs...

[dr-orlovsky]

I have a vague plan to use https://github.com/LNP-BP/bp-node (rust-bitcoin based), once it will be more complete, and embed it into C FFI to be called from iOS Swift.

[BlockMechanic]

A lot of progress was made on the Android front. I keep looking back at this, i hope i can make time and try get the iOS effort on par with android.

[congtungWonder]

@BlockMechanic : Do you have any update at your side?
I'm trying to support iOS right now but still no hope
Please help me to update if you see anything we can work around

[laanwj]

I missed when this was closed. Actually added "Up for grabs" label.

[BlockMechanic]

With the introduction of sqlite, it may be worth while to push this. If i can get a moment i will try this again

[Sjors]

@BlockMechanic I couldn't even get the node to work without a wallet, so Sqlite wasn't the bottleneck.

[CaptThomas]

Just a thought, but what if you just opened the QT project on like MacOS, got QT5 to convert it to an XCode project and see if that will work? Just a thought, let me know if anyone has tested.

[fanquake]

Removing Up for grabs, because it's not clear anyone could take over this changeset. We already support building for arm64-apple-darwin (which is the same target as being added here). I think discussion can continue in #11720 to determine what actually needs to be done.