Darwin builds (server/plugin)

[tiero]

Would be nice to have officially built bwt electrum plugin for MacOSX, without having to download rust toolchain, especially for normies. Thanks

[bitnano]

+1

[shesek]

The options for supporting macOS are: (1) finding a trusted maintainer to help build and publish the macOS releases, (2) getting myself a macOS, or (3) trusting some cloud provider to build it remotely.

I wouldn't feel comfortable trusting a cloud provider for this, so that leaves us with finding a trusted maintainer, or someone donating to sponsor a macOS that I could use for that (which would also enable me to make iOS releases for Spark 😁)

[shesek]

You can also setup the bwt electrum server using docker instead of using the plugin.

This is somewhat more complicated, but not terribly so. Basically docker run --net host -v ~/.bitcoin:/bitcoin shesek/bwt:electrum --xpub <xpub> and you should have the electrum server up and running.

[leshik]

@shesek I was able to successfully cross-compile for macOS in Docker using https://github.com/joseluisq/rust-linux-darwin-builder
Could this be integrated into the release script? Should work on CI too. It would be nice to have signed binaries for macOS.

[shesek]

@leshik Nice, this makes things much easier! I was under the impression that you need a mac for this. Is osxcross considered stable? Does it have any disadvantages compared to a real macOS host? Does it support signing too?

I'm a bit wary about introducing third party docker images into the build process, but its nothing that some auditing and replicating the dockerfile into this repo couldn't solve.

[leshik]

@shesek Unfortunately I don't have answers as I stumbled upon osxcross yesterday just out of curiosity while looking for a solution to compile bwt in Docker.

Before I found this darwin builder Docker image, I was able to do all the steps manually from the fresh debian:buster-slim (this manual helped me a lot). The main obstacle to automating all the things that I discovered is that macOS SDK needs to be downloaded from somewhere.

One possibility is to fetch the XCode xip installer from Apple, then unpack it and repack SDK as described in osxcross readme. Unfortunately, Apple doesn't allow downloading XCode without login. Somebody has to download it manually anyway (all ~8Gb, of which 48Mb only is needed).

Another option is to fetch pre-packaged SDK from somewhere, but this leads to the same issue of trusting somebody.

[pox]

@shesek I develop mostly on a mac so I'm happy to help with releases. I believe we share a few connections (Tel Aviv) if you want to explore this option offline.

[shesek]

@pox Sounds good, happy to explore this! Being able to do an IRL exchange of release pubkeys is also a big plus.

Are you coming to the bitcoin embassy from time to time? Perhaps we could meet there?

I'm reachable at [email protected] or on freenode (same nickname).

[pox]

@shesek sent an email.

[shesek]

I think I managed to cross-compile for macOS (via oxscross), but don't have a way to actually test this.

Would anyone like to be a guinea pig? 🐹

[pox]

Right here! 👋

[shesek]

Thanks @pox!

The macOS releases are available here: https://github.com/shesek/bwt/releases/tag/osx-rc

I recently also made the builds reproducible. If you'd like, you can verify that checksums against the builds made on Travis CI (which pulls the code from GitHub). Scroll to the very end of the build log and you'll see the checksums following -----BEGIN SHA256SUM-----.

[pox]

Doesn't seem to quite work.

When I try executing the bwt binary:

dyld: Symbol not found: _rustsecp256k1_v0_1_1_context_no_precomp

😞

@shesek I see you're using Travis. Have you tried executing the binary as part of the build? Could probably be used as some basic smoke test.

[shesek]

Ugh. I've been running into similar errors in the complication step, I thought I solved them with some compiler flags but it looks like I just pushed it away to the runtime. :(

I'm using Travis, but it's also cross-compiling from Linux environment (to match the binaries produced by my local environment), so I can't run the binary as part of the build. I can, however, create a separate macOS Travis environment specifically for that purpose -- which is an excellent idea. Thanks for bringing that up!

I'll comment here when I make progress. Cheers.

[shesek]

Ok, I think I got it working for realz this time. :) Care to give it a try @pox? (the binaries are available on that same osx-rc tag)

[pox]

@shesek binary seems to work. Is there any sort of acceptance test you'd like me to perform? Or are you just happy with knowing it's running (spitting out usage, for example)?

$ ./bwt
 ERROR bwt::hd > Please provide at least one xpub to track (via --xpub or --bare-xpub).
Error: no xpubs provided

[shesek]

@pox Nice, thanks!

Just seeing that its running without crashing down is probably good enough, but it would be cool if you have an handy bitcoin node and could give this a more throughout test.

Basically, if you have bitcoind running with the default directory and ports, ./bwt --xpub <xpub> should Just Work (tm). Otherwise you'll also also need to set --bitcoind-url and --bitcoind-dir. Also recommended that you use a separate bitcoind wallet (bitcoin-cli createwallet <name> true and --bitcoind-wallet <name>).

After it finishes rescanning, you can try connecting to the electrum server on port 50001 (no ssl, so --server 127.0.0.1:50001:t with the t), or try the http api with something like curl localhost:3060/utxos

[pox]

So far so good, it started a rescan. Will update.

btw, definitely a good idea to have a CLI option to specify rescan block height.

[pox]

@shesek I got Electrum to connect to it, but it didn't find any transactions.
In all likelihood this isn't an issue with the osx build. Probably just a matter of usage. I can elaborate if you want (I've used some known public xpubs that have transactions in the last few months, but they don't show up).

Also, I only tested the standalone binary, not the plugin. Is the plugin also available on osx, and if so would you like me to test it? I would need instructions how.

[shesek]

btw, definitely a good idea to have a CLI option to specify rescan block height.

This is possible via --xpub <xpub>:<height> or using a date with --xpub <xpub>:<yyyy-mm-dd>.

(I know you already figured that out, just mentioning it for others.)

[shesek]

Implemented in bc9162a.

[shesek]

Released in v0.1.5! https://github.com/shesek/bwt/releases/tag/v0.1.5