Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

removeReferencesToVendoredSourcesHook uses 18GB of memory #744

Closed
DaRacci opened this issue Nov 22, 2024 · 2 comments · Fixed by #745
Closed

removeReferencesToVendoredSourcesHook uses 18GB of memory #744

DaRacci opened this issue Nov 22, 2024 · 2 comments · Fixed by #745
Labels
bug Something isn't working

Comments

@DaRacci
Copy link

DaRacci commented Nov 22, 2024 •
edited
Loading

Describe the bug
When compiling any crate for any target, be it native or cross compiled when the installPhase is ran and removeReferencesToVendoredSourcesHook is running i seed sed sleep for about 40 seconds, followed by a rapid jump in memory usage up to around 18GB.

image

I've tried messing around by extracting the script and running it manually but cannot recreate without it being part of the build.
Also worth noting I'm using crane through dream2nix which I'm using through nix-cargo-integration.

Reproduction
The Repository is located at https://github.com/AMTSupport/tools
This is a monorepo with multiple programs the issue occurs in all of them so far in my testing but for reference you can run nix build .#cleaner-x86_64-linux-dev

Logs

cleaner-deps> cargoArtifacts not set, will not reuse any cargo artifacts
cleaner-deps> Running phase: unpackPhase
cleaner-deps> dream2nix: installing cargo vendor directory from /nix/store/lp9hshc5vcz7nam55nb4nnzk5l4d5q6p-vendor to /build/nix-vendor
cleaner-deps> unpacking source archive /nix/store/6z8da5w3jwzw9ndk1a36xmz3hparqbml-source
cleaner-deps> source root is source
cleaner-deps> Running phase: patchPhase
cleaner-deps> moving Cargo.lock to Cargo.lock.orig, then will use /nix/store/cjdp7qv9n45rlbb2f29n8knf89g54343-Cargo.lock as Cargo.lock
cleaner-deps> Executing configureCargoCommonVars
cleaner-deps> Running phase: updateAutotoolsGnuConfigScriptsPhase
cleaner-deps> Running phase: configurePhase
cleaner-deps> dream2nix: Writing git vendor entries to /build/.cargo_home/config.toml
cleaner-deps> dream2nix: replacing relative dependency paths with absolute paths in Cargo.toml
cleaner-deps> setting source replacement config in /build/.cargo_home/config.toml using vendored directory /build/nix-vendor
cleaner-deps> default configurePhase, nothing to do
cleaner-deps> Running phase: buildPhase
cleaner-deps> ++ command cargo --version
cleaner-deps> cargo 1.84.0-nightly (4a2d8dc63 2024-11-09)
cleaner-deps> ++ command cargo check --profile dev --package cleaner --locked
cleaner-deps> warning: /build/source/crates/rebooter/Cargo.toml: `default-features` is ignored for chrono, since `default-features` was not specified for `workspace.dependencies.chrono`, this could become a hard error in the future
cleaner-deps>    Compiling proc-macro2 v1.0.86
cleaner-deps>    Compiling unicode-ident v1.0.13
cleaner-deps>    Compiling libc v0.2.159
cleaner-deps>    Compiling autocfg v1.4.0
cleaner-deps>    Compiling version_check v0.9.5
cleaner-deps>     Checking log v0.4.22
cleaner-deps>     Checking once_cell v1.20.1
cleaner-deps>     Checking utf8parse v0.2.2
cleaner-deps>     Checking cfg-if v1.0.0
cleaner-deps>    Compiling crossbeam-utils v0.8.20
cleaner-deps>    Compiling syn v1.0.109
cleaner-deps>     Checking smallvec v1.13.2
cleaner-deps>     Checking lazy_static v1.5.0
cleaner-deps>     Checking itoa v1.0.11
cleaner-deps>     Checking unicode-width v0.1.14
cleaner-deps>    Compiling parking_lot_core v0.9.10
cleaner-deps>    Compiling signal-hook v0.3.17
cleaner-deps>     Checking overload v0.1.1
cleaner-deps>     Checking colorchoice v1.0.2
cleaner-deps>     Checking is_terminal_polyfill v1.70.1
cleaner-deps>     Checking scopeguard v1.2.0
cleaner-deps>     Checking anstyle v1.0.8
cleaner-deps>     Checking anstyle-query v1.1.1
cleaner-deps>     Checking arrayvec v0.7.6
cleaner-deps>     Checking unicode-segmentation v1.12.0
cleaner-deps>     Checking anstyle-parse v0.2.5
cleaner-deps>    Compiling thiserror v1.0.64
cleaner-deps>    Compiling portable-atomic v1.9.0
cleaner-deps>    Compiling rustix v0.38.37
cleaner-deps>     Checking pin-project-lite v0.2.14
cleaner-deps>     Checking nu-ansi-term v0.46.0
cleaner-deps>     Checking sharded-slab v0.1.7
cleaner-deps>     Checking powerfmt v0.2.0
cleaner-deps>     Checking strsim v0.11.1
cleaner-deps>    Compiling anyhow v1.0.89
cleaner-deps>     Checking tracing-core v0.1.32
cleaner-deps>     Checking thread_local v1.1.8
cleaner-deps>    Compiling heck v0.5.0
cleaner-deps>     Checking bitflags v2.6.0
cleaner-deps>     Checking clap_lex v0.7.2
cleaner-deps>     Checking anstream v0.6.15
cleaner-deps>     Checking linux-raw-sys v0.4.14
cleaner-deps>     Checking deranged v0.3.11
cleaner-deps>     Checking num-conv v0.1.0
cleaner-deps>     Checking fastrand v2.1.1
cleaner-deps>    Compiling rayon-core v1.12.1
cleaner-deps>     Checking time-core v0.1.2
cleaner-deps>    Compiling paste v1.0.15
cleaner-deps>     Checking bitflags v1.3.2
cleaner-deps>     Checking number_prefix v0.4.0
cleaner-deps>     Checking newline-converter v0.3.0
cleaner-deps>     Checking clap_builder v4.5.18
cleaner-deps>    Compiling macros v0.2.0 (/build/source/crates/macros)
cleaner-deps>    Compiling proc-macro-error-attr v1.0.4
cleaner-deps>    Compiling proc-macro-error v1.0.4
cleaner-deps>    Compiling lib v0.2.0 (/build/source/crates/lib)
cleaner-deps>     Checking dyn-clone v1.0.17
cleaner-deps>     Checking iana-time-zone v0.1.61
cleaner-deps>     Checking tracing-log v0.2.0
cleaner-deps>     Checking futures-core v0.3.30
cleaner-deps>     Checking is_executable v1.0.3
cleaner-deps>     Checking sysexits v0.8.2
cleaner-deps>     Checking either v1.13.0
cleaner-deps>     Checking impls v1.0.3
cleaner-deps>     Checking shlex v1.3.0
cleaner-deps>    Compiling cleaner v0.2.0 (/build/source/crates/cleaner)
cleaner-deps>     Checking glob v0.3.1
cleaner-deps>    Compiling lock_api v0.4.12
cleaner-deps>    Compiling num-traits v0.2.19
cleaner-deps>     Checking downcast-rs v1.2.1
cleaner-deps>     Checking tracing-subscriber v0.3.18
cleaner-deps>     Checking time v0.3.36
cleaner-deps>     Checking quote v1.0.37
cleaner-deps>     Checking crossbeam-epoch v0.9.18
cleaner-deps>     Checking crossbeam-channel v0.5.13
cleaner-deps>     Checking syn v2.0.79
cleaner-deps>     Checking crossbeam-deque v0.8.5
cleaner-deps>     Checking signal-hook-registry v1.4.2
cleaner-deps>     Checking mio v0.8.11
cleaner-deps>     Checking console v0.15.8
cleaner-deps>     Checking permissions v0.5.1
cleaner-deps>    Compiling vte_generate_state_changes v0.1.2
cleaner-deps>     Checking parking_lot v0.12.3
cleaner-deps>     Checking signal-hook-mio v0.2.4
cleaner-deps>     Checking crossterm v0.25.0
cleaner-deps>     Checking chrono v0.4.38
cleaner-deps>     Checking rayon v1.10.0
cleaner-deps>     Checking vte v0.11.1
cleaner-deps>     Checking vt100 v0.15.2
cleaner-deps>     Checking indicatif v0.17.8
cleaner-deps>     Checking tempfile v3.13.0
cleaner-deps>    Compiling quote-use v0.5.1
cleaner-deps>    Compiling attribute-derive-macro v0.3.1
cleaner-deps>    Compiling thiserror-impl v1.0.64
cleaner-deps>    Compiling clap_derive v4.5.18
cleaner-deps>    Compiling tracing-attributes v0.1.27
cleaner-deps>    Compiling tokio-macros v2.4.0
cleaner-deps>    Compiling async-trait v0.1.83
cleaner-deps>    Compiling attribute-derive v0.3.1
cleaner-deps>     Checking tokio v1.40.0
cleaner-deps>    Compiling michie-macro v3.0.2
cleaner-deps>     Checking inquire v0.6.2 (https://github.com/anwarhahjjeffersongeorge/inquire-filepath/?branch=filepath-select#a5f53ecb)
cleaner-deps>     Checking tracing-appender v0.2.3
cleaner-deps>     Checking tracing v0.1.40
cleaner-deps>     Checking tracing-indicatif v0.3.6
cleaner-deps>     Checking michie v3.0.2
cleaner-deps>     Checking clap v4.5.18
cleaner-deps>     Checking clap_complete v4.5.29
cleaner-deps>     Checking tokio-stream v0.1.16
cleaner-deps>     Finished dev profile [unoptimized + debuginfo] target(s) in 6.72s
cleaner-deps> ++ command cargo build --profile dev --package cleaner --locked
cleaner-deps> warning: /build/source/crates/rebooter/Cargo.toml: `default-features` is ignored for chrono, since `default-features` was not specified for `workspace.dependencies.chrono`, this could become a hard error in the future
cleaner-deps>    Compiling log v0.4.22
cleaner-deps>    Compiling cfg-if v1.0.0
cleaner-deps>    Compiling utf8parse v0.2.2
cleaner-deps>    Compiling once_cell v1.20.1
cleaner-deps>    Compiling smallvec v1.13.2
cleaner-deps>    Compiling lazy_static v1.5.0
cleaner-deps>    Compiling itoa v1.0.11
cleaner-deps>    Compiling unicode-width v0.1.14
cleaner-deps>    Compiling arrayvec v0.7.6
cleaner-deps>    Compiling is_terminal_polyfill v1.70.1
cleaner-deps>    Compiling overload v0.1.1
cleaner-deps>    Compiling colorchoice v1.0.2
cleaner-deps>    Compiling scopeguard v1.2.0
cleaner-deps>    Compiling anstyle-query v1.1.1
cleaner-deps>    Compiling anstyle-parse v0.2.5
cleaner-deps>    Compiling libc v0.2.159
cleaner-deps>    Compiling crossbeam-utils v0.8.20
cleaner-deps>    Compiling anstyle v1.0.8
cleaner-deps>    Compiling unicode-segmentation v1.12.0
cleaner-deps>    Compiling pin-project-lite v0.2.14
cleaner-deps>    Compiling sharded-slab v0.1.7
cleaner-deps>    Compiling lock_api v0.4.12
cleaner-deps>    Compiling nu-ansi-term v0.46.0
cleaner-deps>    Compiling tracing-core v0.1.32
cleaner-deps>    Compiling thread_local v1.1.8
cleaner-deps>    Compiling bitflags v2.6.0
cleaner-deps>    Compiling strsim v0.11.1
cleaner-deps>    Compiling unicode-ident v1.0.13
cleaner-deps>    Compiling clap_lex v0.7.2
cleaner-deps>    Compiling powerfmt v0.2.0
cleaner-deps>    Compiling linux-raw-sys v0.4.14
cleaner-deps>    Compiling vte v0.11.1
cleaner-deps>    Compiling thiserror v1.0.64
cleaner-deps>    Compiling portable-atomic v1.9.0
cleaner-deps>    Compiling number_prefix v0.4.0
cleaner-deps>    Compiling bitflags v1.3.2
cleaner-deps>    Compiling num-conv v0.1.0
cleaner-deps>    Compiling time-core v0.1.2
cleaner-deps>    Compiling fastrand v2.1.1
cleaner-deps>    Compiling proc-macro2 v1.0.86
cleaner-deps>    Compiling anstream v0.6.15
cleaner-deps>    Compiling anyhow v1.0.89
cleaner-deps>    Compiling dyn-clone v1.0.17
cleaner-deps>    Compiling tokio v1.40.0
cleaner-deps>    Compiling deranged v0.3.11
cleaner-deps>    Compiling num-traits v0.2.19
cleaner-deps>    Compiling sysexits v0.8.2
cleaner-deps>    Compiling is_executable v1.0.3
cleaner-deps>    Compiling iana-time-zone v0.1.61
cleaner-deps>    Compiling either v1.13.0
cleaner-deps>    Compiling shlex v1.3.0
cleaner-deps>    Compiling futures-core v0.3.30
cleaner-deps>    Compiling impls v1.0.3
cleaner-deps>    Compiling newline-converter v0.3.0
cleaner-deps>    Compiling michie v3.0.2
cleaner-deps>    Compiling clap_builder v4.5.18
cleaner-deps>    Compiling downcast-rs v1.2.1
cleaner-deps>    Compiling glob v0.3.1
cleaner-deps>    Compiling tracing-log v0.2.0
cleaner-deps>    Compiling tracing v0.1.40
cleaner-deps>    Compiling crossbeam-epoch v0.9.18
cleaner-deps>    Compiling crossbeam-channel v0.5.13
cleaner-deps>    Compiling rustix v0.38.37
cleaner-deps>    Compiling signal-hook-registry v1.4.2
cleaner-deps>    Compiling mio v0.8.11
cleaner-deps>    Compiling parking_lot_core v0.9.10
cleaner-deps>    Compiling console v0.15.8
cleaner-deps>    Compiling permissions v0.5.1
cleaner-deps>    Compiling quote v1.0.37
cleaner-deps>    Compiling tracing-subscriber v0.3.18
cleaner-deps>    Compiling crossbeam-deque v0.8.5
cleaner-deps>    Compiling signal-hook v0.3.17
cleaner-deps>    Compiling parking_lot v0.12.3
cleaner-deps>    Compiling syn v2.0.79
cleaner-deps>    Compiling rayon-core v1.12.1
cleaner-deps>    Compiling vt100 v0.15.2
cleaner-deps>    Compiling signal-hook-mio v0.2.4
cleaner-deps>    Compiling chrono v0.4.38
cleaner-deps>    Compiling time v0.3.36
cleaner-deps>    Compiling crossterm v0.25.0
cleaner-deps>    Compiling rayon v1.10.0
cleaner-deps>    Compiling indicatif v0.17.8
cleaner-deps>    Compiling tracing-indicatif v0.3.6
cleaner-deps>    Compiling tokio-stream v0.1.16
cleaner-deps>    Compiling tracing-appender v0.2.3
cleaner-deps>    Compiling clap v4.5.18
cleaner-deps>    Compiling clap_complete v4.5.29
cleaner-deps>    Compiling macros v0.2.0 (/build/source/crates/macros)
cleaner-deps>    Compiling tempfile v3.13.0
cleaner-deps>    Compiling inquire v0.6.2 (https://github.com/anwarhahjjeffersongeorge/inquire-filepath/?branch=filepath-select#a5f53ecb)
cleaner-deps>    Compiling lib v0.2.0 (/build/source/crates/lib)
cleaner-deps>    Compiling cleaner v0.2.0 (/build/source/crates/cleaner)
cleaner-deps>     Finished dev profile [unoptimized + debuginfo] target(s) in 3.94s
cleaner-deps> Running phase: installPhase
cleaner-deps> no previous artifacts found, compressing and installing full archive of target to /nix/store/b2yhv7f2sbc9vang102s1lrpcfywb537-cleaner-deps-0.2.0/target.tar.zst
cleaner-deps> /*stdin*\            : 24.14%   (   539 MiB =>    130 MiB, /nix/store/b2yhv7f2sbc9vang102s1lrpcfywb537-cleaner-deps-0.2.0/target.tar.zst)
cleaner-deps> Running phase: fixupPhase
cleaner-deps> shrinking RPATHs of ELF executables and libraries in /nix/store/b2yhv7f2sbc9vang102s1lrpcfywb537-cleaner-deps-0.2.0
cleaner-deps> checking for references to /build/ in /nix/store/b2yhv7f2sbc9vang102s1lrpcfywb537-cleaner-deps-0.2.0...
cleaner-deps> patching script interpreter paths in /nix/store/b2yhv7f2sbc9vang102s1lrpcfywb537-cleaner-deps-0.2.0
cleaner-deps> fixupPhase completed in 59 seconds
cleaner> Running phase: unpackPhase
cleaner> dream2nix: installing cargo vendor directory from /nix/store/b2yhv7f2sbc9vang102s1lrpcfywb537-cleaner-deps-0.2.0/nix-vendor to /build/nix-vendor
cleaner> unpacking source archive /nix/store/m9i2kaki0n40jfy0qapsw7y1a4v6mqvp-d903gg4rksis9rmj2aa3ks2pv1aqbmqq-source
cleaner> source root is d903gg4rksis9rmj2aa3ks2pv1aqbmqq-source
cleaner> Running phase: patchPhase
cleaner> Executing configureCargoCommonVars
cleaner> decompressing cargo artifacts from /nix/store/b2yhv7f2sbc9vang102s1lrpcfywb537-cleaner-deps-0.2.0/target.tar.zst to target
cleaner> Running phase: updateAutotoolsGnuConfigScriptsPhase
cleaner> Running phase: configurePhase
cleaner> dream2nix: Writing git vendor entries to /build/.cargo_home/config.toml
cleaner> dream2nix: replacing relative dependency paths with absolute paths in Cargo.toml
cleaner> dream2nix: replacing Cargo.lock with /nix/store/cjdp7qv9n45rlbb2f29n8knf89g54343-Cargo.lock
cleaner> setting source replacement config in /build/.cargo_home/config.toml using vendored directory /build/nix-vendor
cleaner> default configurePhase, nothing to do
cleaner> Running phase: buildPhase
cleaner> ++ command cargo --version
cleaner> cargo 1.84.0-nightly (4a2d8dc63 2024-11-09)
cleaner> ++ command cargo build --profile dev --package cleaner --message-format json-render-diagnostics --locked
cleaner> warning: /build/d903gg4rksis9rmj2aa3ks2pv1aqbmqq-source/crates/rebooter/Cargo.toml: default-features is ignored for chrono, since default-features was not specified for workspace.dependencies.chrono, this could become a hard error in the future
cleaner>    Compiling anyhow v1.0.89
cleaner>    Compiling impls v1.0.3
cleaner>    Compiling lib v0.2.0 (/build/d903gg4rksis9rmj2aa3ks2pv1aqbmqq-source/crates/lib)
cleaner>    Compiling macros v0.2.0 (/build/d903gg4rksis9rmj2aa3ks2pv1aqbmqq-source/crates/macros)
cleaner>    Compiling cleaner v0.2.0 (/build/d903gg4rksis9rmj2aa3ks2pv1aqbmqq-source/crates/cleaner)
cleaner>     Finished dev profile [unoptimized + debuginfo] target(s) in 2.96s
cleaner> Running phase: installPhase
cleaner> searching for bins/libs to install from cargo build log at cargoBuildLog01OC.json
cleaner> installing /build/d903gg4rksis9rmj2aa3ks2pv1aqbmqq-source/target/x86_64-unknown-linux-gnu/debug/cleaner
cleaner> searching for bins/libs complete
cleaner> stripping references to cargoVendorDir from:
cleaner> /nix/store/nzhc0c9rcsrsx67v3dasg2gzxzfvk4gn-cleaner-0.2.0/bin/cleaner
cleaner> stripping references done
cleaner> installPhase completed in 1 minutes 13 seconds
cleaner> Running phase: fixupPhase
cleaner> shrinking RPATHs of ELF executables and libraries in /nix/store/nzhc0c9rcsrsx67v3dasg2gzxzfvk4gn-cleaner-0.2.0
cleaner> shrinking /nix/store/nzhc0c9rcsrsx67v3dasg2gzxzfvk4gn-cleaner-0.2.0/bin/cleaner
cleaner> checking for references to /build/ in /nix/store/nzhc0c9rcsrsx67v3dasg2gzxzfvk4gn-cleaner-0.2.0...
cleaner> patching script interpreter paths in /nix/store/nzhc0c9rcsrsx67v3dasg2gzxzfvk4gn-cleaner-0.2.0
cleaner> stripping (with command strip and flags -S -p) in  /nix/store/nzhc0c9rcsrsx67v3dasg2gzxzfvk4gn-cleaner-0.2.0/bin
@DaRacci DaRacci added the bug Something isn't working label Nov 22, 2024
@ipetkov ipetkov mentioned this issue Nov 23, 2024
Merged
4 tasks
@ipetkov
Copy link
Owner

ipetkov commented Nov 23, 2024

Hi @DaRacci thanks for the report! Should be fixed with #745

@DaRacci
Copy link
Author

DaRacci commented Nov 25, 2024

Confirming this has resolved the issue after testing, its also sped up the action by about 2x.

DaRacci added a commit to AMTSupport/tools that referenced this issue Nov 25, 2024
@DaRacci
chore(deps): Update crane to fix ipetkov/crane#744
35b27de
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

removeReferencesToVendoredSourcesHook: dedup references ipetkov/crane
2 participants
@ipetkov @DaRacci