build: set up reproducible builds via Docker #206
Conversation
| @@ -1,6 +1,15 @@ | |||
| use cketh_common::eth_rpc_client::providers::RpcApi; | |||
|
|
|||
| use crate::*; | |||
| use crate::{ | |||
There was a problem hiding this comment.
This refactoring was necessary for the two-stage Rust build (--only-dependencies and --evm-rpc) to work as expected in the Docker environment.
| apt -yq update && \ | ||
| apt -yqq install --no-install-recommends curl ca-certificates \ | ||
| build-essential pkg-config libssl-dev llvm-dev liblmdb-dev clang cmake |
There was a problem hiding this comment.
This comes from the Internet Identity Dockerfile. Does this remain reproducible if one of these packages is updated on the apt registry?
There was a problem hiding this comment.
That's a good question, maybe @frederikrothenberger can enlighten us here 😁
There was a problem hiding this comment.
We never ran into issues so far, but good point. It seems broken?
@nmattia: did we just get lucky (or didn't notice)?
There was a problem hiding this comment.
We got lucky! And in general Ubuntu doesn't do big updates without an actual Ubuntu update; stable enough for the needs of II
There was a problem hiding this comment.
just to clarify the context for II (I have no context here): II pins cargo & npm which are the two important tools; and npm statically links most libraries that may cause differences in the build output (zlib, etc)
There was a problem hiding this comment.
Thanks a lot guys for the quick answer! If it's good enough for II, it should be good enough for the EVM RPC canister.
There was a problem hiding this comment.
ok, looks like all the bash files were taken from II. If reproducibility is important you might also want to copy the repro tests that run on CI:
- https://github.com/dfinity/internet-identity/blob/c31e8e1c2cd0ba2e9b9fa2a5b93f2e26d1d0a433/.github/workflows/canister-tests.yml#L901-L960
- https://github.com/dfinity/internet-identity/blob/main/.github/workflows/release-build-check.yml
and if the apt issue actually becomes an issue you're probably better off switching to bazel or nix, but you should be fine
There was a problem hiding this comment.
This is a problem shared by II, cycles-ledger and now the EVM RPC canister. Eventually we need to address it.
There was a problem hiding this comment.
Thanks a lot @rvanasa for this effort. I think the pieces are mostly there; however, I failed to reproduce the build between Mac and devenv, which is unexpected. I think the CI checks suggested by Nicolas would make a lot of sense.
| apt -yq update && \ | ||
| apt -yqq install --no-install-recommends curl ca-certificates \ | ||
| build-essential pkg-config libssl-dev llvm-dev liblmdb-dev clang cmake |
There was a problem hiding this comment.
Thanks a lot guys for the quick answer! If it's good enough for II, it should be good enough for the EVM RPC canister.
Dockerfile
Outdated
| && rm -rf src \ | ||
| && rm -rf Cargo.toml \ | ||
| && rm -rf Cargo.lock \ | ||
| && rm -rf target \ |
There was a problem hiding this comment.
Are you sure about this? AFAIK, II put the built dependencies in
ENV CARGO_TARGET_DIR=/cargo_target
and never deletes it. Otherwise I currently don't see the point of pre-building the cargo dependencies.
There was a problem hiding this comment.
Removed rm -rf target; the other lines are necessary for the COPY line below to work as expected.
| CARGO_HOME=/cargo \ | ||
| PATH=/cargo/bin:$PATH | ||
|
|
||
| WORKDIR /evm_rpc |
There was a problem hiding this comment.
unless I'm missing something, I don't think you need an explicit working directory, the default (I think /) should be fine
There was a problem hiding this comment.
This is a workaround for a nebulous error message (along the lines of ERROR: cannot copy to non-directory: /var/lib/docker/overlay2/fso8ezwpr886l9bjnx3jni37e/merged/lib). I'll address this in a follow-up PR (trying to get this merged by the end of the day if possible).
| RUN mkdir -p src \ | ||
| && echo "fn main() {}" > src/main.rs \ | ||
| && echo "" > src/lib.rs \ | ||
| && mkdir -p e2e/rust/src \ |
There was a problem hiding this comment.
Since we only care about building the canister evm_rpc.wasm.gz, why do we need to care about stuff in e2e?
There was a problem hiding this comment.
This was required to allow the Cargo workspace to pre-build as expected (since it also refers to the E2E Cargo.toml file).
Co-authored-by: gregorydemay <[email protected]>
Co-authored-by: gregorydemay <[email protected]>
|
LGTM, thanks for pushing this effort @rvanasa! |
This PR enables reproducible builds via a Docker container. Based on the approach used for the Internet Identity canister.
Supersedes #193.