thread 'main' panicked at 'rustc not found in /scratch/rustc/build/sparcv9/build/bootstrap/debug/deps, run cargo build --bins before cargo run', lib.rs:415:13

[psumbera]

Rust 1.70 build on Solaris sometimes files on some machines:

Building tool rustfmt (stage2 -> stage3)
   Compiling cc v1.0.77
   Compiling camino v1.0.9
   Compiling rustfmt-nightly v1.5.2 (/scratch/psumbera/userland-test/components/rust/rustc/rustc-1.70.0-src/src/tools/rustfmt)
   Compiling yansi-term v0.1.2
   Compiling getopts v0.2.21
   Compiling unicode_categories v0.1.1
   Compiling unicode-segmentation v1.10.0
   Compiling diff v0.1.13
   Compiling bytecount v0.6.2
   Compiling dirs-sys-next v0.1.2
   Compiling dirs-sys v0.3.6
   Compiling rustfmt-config_proc_macro v0.3.0 (/scratch/psumbera/userland-test/components/rust/rustc/rustc-1.70.0-src/src/tools/rustfmt/config_proc_macro)
   Compiling derive-new v0.5.8
   Compiling env_logger v0.9.0
   Compiling cargo-platform v0.1.2
   Compiling toml v0.5.7
   Compiling dirs-next v2.0.0
   Compiling dirs v4.0.0
   Compiling annotate-snippets v0.9.1
   Compiling term v0.7.0
   Compiling cargo_metadata v0.14.0
   Compiling libz-sys v1.1.3
   Compiling rustc-workspace-hack v1.0.0 (/scratch/psumbera/userland-test/components/rust/rustc/rustc-1.70.0-src/src/tools/rustc-workspace-hack)
    Finished release [optimized] target(s) in 2m 41s
Build completed successfully in 2:11:14
Building bootstrap
    Finished dev [unoptimized] target(s) in 1.38s
warning: file locking not supported for target, not locking build directory
thread 'main' panicked at '`rustc` not found in /scratch/psumbera/userland-test/components/rust/rustc/build/sparcv9/build/bootstrap/debug/deps, run `cargo build --bins` before `cargo run`', lib.rs:415:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:00:02
make[1]: *** [Makefile:13: all] Error 1
make[1]: Leaving directory '/scratch/psumbera/userland-test/components/rust/rustc/build/sparcv9'
'/usr/gnu/bin/make' failed with error code 2!
Running a cleanup program to restore the state.
gmake[1]: Entering directory '/scratch/psumbera/userland-test/components/rust/rustc/build/sparcv9'
Building bootstrap
    Finished dev [unoptimized] target(s) in 1.25s
warning: file locking not supported for target, not locking build directory
thread 'main' panicked at '`rustc` not found in /scratch/psumbera/userland-test/components/rust/rustc/build/sparcv9/build/bootstrap/debug/deps, run `cargo build --bins` before `cargo run`', lib.rs:415:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:00:01
gmake[1]: *** [Makefile:25: clean] Error 1
gmake[1]: Leaving directory '/scratch/psumbera/userland-test/components/rust/rustc/build/sparcv9'
Failed to run cleanup program, terminating prematurely...

I wonder whether it can do something with: warning: file locking not supported for target, not locking build directory?!

[jyn514]

are you running make in parallel? that's not supported on solaris (you made the PR removing the build locking IIRC).

[psumbera]

I see it does: /usr/gnu/bin/make -j 16 -l 32.

I think I should look once again if it's possible make fd-lock to use fcntl locking (as it's now in `rustix...

You can close the bug. Thank you!

[psumbera]

Adding reference to main proble - fd-lock isn't supported on Solaris due missing flock() syscall).
#103630

[psumbera]

@jyn514 I'm confused now. I have seen the same error even with make -j 1 (just one ocurance, not two as mentioned above).

Also tried to dig closely how locking works and put into fd-lock some debug prints with process id and lock file name. But later when I was inspecting still running process it didn't have any lock file open. No file descriptor no lock?!

[jyn514]

@psumbera did you clear the cache before running make -j1? if the build directory gets into an inconsistent state we don't recover it automatically.

i don't know what's going on with fd-lock, but the file that bootstrap locks is build/lock.

[psumbera]

@psumbera did you clear the cache before running make -j1? if the build directory gets into an inconsistent state we don't recover it automatically.

Everything is removed before new build...

[jyn514]

exactly what command are you running and what is your config.toml?

[psumbera]

i don't know what's going on with fd-lock, but the file that bootstrap locks is build/lock.

If I see it right, lock file is created/opened here:

rust/src/bootstrap/bin/main.rs

Line 25 in 536635c

build_lock = fd_lock::RwLock::new(t!(std::fs::File::create(&path)));

and the file descriptor for the lock file is closed on block exit here:

rust/src/bootstrap/bin/main.rs

Line 38 in 536635c

}

Isn't it mean that locking isn't working even on Linux?

[jyn514]

no, it's declared outside the block

rust/src/bootstrap/bin/main.rs

Line 21 in 536635c

let _build_lock_guard;

[psumbera]

no, it's declared outside the block

rust/src/bootstrap/bin/main.rs

Line 21 in 536635c

let _build_lock_guard;

Right. Unfortunatelly version 1.70.0 hasn't fix for this (my) regression. And the declaration is still insde the block.

[jyn514]

we do not support old versions of rust. the only supported release is the latest stable release.

[psumbera]

we do not support old versions of rust. the only supported release is the latest stable release.

Isn't version 1.70.0 latest stable version? I'm just curious.

[jyn514]

yes, you're right. i'm sorry.

i think you should ask someone else for help on this issue. i am having trouble focusing today.

[psumbera]

exactly what command are you running and what is your config.toml?

grep -v ^# build/amd64/config.toml
changelog-seen = 2

[llvm]

[build]

cargo = '/builds/psumbera/rustc-1.70.0/bin/cargo'

rustc = '/builds/psumbera/rustc-1.70.0/bin/rustc'

extended = true

tools = ["cargo", "rustfmt"]

configure-args = ['--prefix=/usr', '--libdir=/usr/lib/amd64', '--default-linker=gcc', '--enable-local-rust']

[install]

prefix = '/usr'

libdir = '/usr/lib/amd64'

[rust]

default-linker = 'gcc'

[target.x86_64-pc-solaris]

[dist]

I run:
configure --prefix=/usr --libdir=/usr/lib/amd64 --default-linker=gcc --enable-local-rust

and then:
make -j 16 -l 32

Where on some systems helps to run -j 1 while on others it breaks it.

rust/src/bootstrap/lib.rs

Line 416 in 90c5418

"`rustc` not found in {}, run `cargo build --bins` before `cargo run`",

When I look into the directory where rustc should be present:

ls -1 build/amd64/build/bootstrap/debug/deps/rustc*
build/amd64/build/bootstrap/debug/deps/rustc-ccd235d729be7b87
build/amd64/build/bootstrap/debug/deps/rustc-ccd235d729be7b87.d

[jyn514]

i'm not sure why this is looking in debug/deps. can you verify that build/amd64/build/bootstrap/debug/rustc exists?

i wonder if bootstrap/debug/bootstrap is a hard-link to bootstrap/debug/deps/bootstrap and that confuses current_exe on solaris.

[psumbera]

i'm not sure why this is looking in debug/deps. can you verify that build/amd64/build/bootstrap/debug/rustc exists?

It does:

ls -l build/amd64/build/bootstrap/debug/rustc
-rwxr-xr-x   2 psumbera staff    3295720 Jun 23 00:41 build/amd64/build/bootstrap/debug/rustc

i wonder if bootstrap/debug/bootstrap is a hard-link to bootstrap/debug/deps/bootstrap and that confuses current_exe on solaris.

It's hardling.

When I remove the asserion the build passes.

[psumbera]

When I remove the asserion the build passes.

It wasn't true. It just didnt' hit the problem when the code was rebuilt.

Now I'm trying following:

--- rustc-1.70.0-src/src/bootstrap/lib.rs
+++ rustc-1.70.0-src/src/bootstrap/lib.rs
@@ -405,11 +405,17 @@
             .expect("failed to read src/version");
         let version = version.trim();

-        let bootstrap_out = std::env::current_exe()
+        let mut bootstrap_out = std::env::current_exe()
             .expect("could not determine path to running process")
             .parent()
             .unwrap()
             .to_path_buf();
+        // Since boostrap is hardlink to deps/boostrap-*, Solaris can sometimes give
+        // us path with deps/ which is bad and need to be avoided.
+        if  bootstrap_out.ends_with("deps") {
+           println!("XXX Removing deps! Got: {:?}", bootstrap_out);
+           bootstrap_out.pop();
+        }
         if !bootstrap_out.join(exe("rustc", config.build)).exists() && !cfg!(test) {
             // this restriction can be lifted whenever https://github.com/rust-lang/rfcs/pull/3028 is implemented
             panic!(

It must execute sometime deps/bootstrap-[hash] instead of just bootstrap.