bootstrap: Build with `use-lld = true` fails under MSYS2 due to some path escaping issues #137498

petrochenkov · 2025-02-23T21:27:59Z

The build fails soon after start with a log like this:

we@WIN-GQ6AS7OLS34 MINGW64 ~/rust
$ ./x t -j1
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.05s
Building stage0 tool tidy (x86_64-pc-windows-gnu)
   Compiling serde_derive v1.0.217
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,C:\\msys64\\tmp\\rustcIGBlzu\\list.def" "-fno-use-linker-plugin" "-Wl,--dynamicbase" "-Wl,--disable-auto-image-base" "-m64" "-Wl,--high-entropy-va" "<sysroot>\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\rsbegin.o" "C:\\msys64\\tmp\\rustcIGBlzu\\symbols
.o" "<16 object files omitted>" "<sysroot>-bootstrap-tools\\release\\deps\\serde_derive-289e03abbb61a8ac.7h71bf3kbtp6y7yvtao8biynu.rcgu.rmeta" "<1 object files omitted>" "-Wl,-Bstatic" "C:\\msys64\\home\\we\\rust\\build\\x86_64-pc-windows-gnu\\stage0-bootstrap-tools\\rel
ease\\deps/{libsyn-c238c3c558a6e659.rlib,libquote-9a71ca2559253843.rlib,libproc_macro2-eff81bcdd40c9193.rlib,libunicode_ident-203f2b90658d2aa5.rlib}.rlib" "<sysroot>\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib/{libproc_macro-*,libstd-*,libpanic_unwind-*,libobject-*,libmemc
hr-*,libaddr2line-*,libgimli-*,libwindows_targets-*,librustc_demangle-*,libstd_detect-*,libhashbrown-*,librustc_std_workspace_alloc-*,libminiz_oxide-*,libadler2-*,libunwind-*,libcfg_if-*,liblibc-*,liballoc-*,librustc_std_workspace_core-*,libcore-*,libcompiler_builtins-*}
.rlib" "-Wl,-Bdynamic" "-lkernel32" "-lkernel32" "-ladvapi32" "-lntdll" "-luserenv" "-lws2_32" "-ldbghelp" "-lgcc_eh" "-l:libpthread.a" "-lmsvcrt" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-lmingwex" "-luser32" "-lkernel32" "-fuse-ld=lld" "-Wl,--nxcompat" "-o" "<sysroo
t>-bootstrap-tools\\release\\deps\\serde_derive-289e03abbb61a8ac.dll" "-Wl,--gc-sections" "-shared" "-Wl,--out-implib=<sysroot>-bootstrap-tools\\release\\deps\\libserde_derive-289e03abbb61a8ac.dll.a" "-nodefaultlibs" "<sysroot>\\lib\\rustlib\\x86_64-pc-windows-gnu\\lib\\
rsend.o"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0librustlibx86_64-pc-windows-gnulibrsbegin.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64tmprustcIGBlzusymbols.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.00.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.01.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.02.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.03.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.04.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.05.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.06.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.07.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.08.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.09.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.10.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.11.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.12.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.13.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.14.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.serde_derive.aa02f1374bc7d472-cgu.15.rcgu.o': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.7h71bf3kbtp6y7yvtao8biynu.rcgu.rmeta': No such file or directory␍
          ld.lld: error: could not open 'C:msys64homewerustbuildx86_64-pc-windows-gnustage0-bootstrap-toolsreleasedepsserde_derive-289e03abbb61a8ac.0g3fdgiuvv97ejvcm8q5w0d3b.rcgu.o': No such file or directory␍
          ld.lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)␍
          collect2.exe: error: ld returned 1 exit status


error: could not compile `serde_derive` (lib) due to 1 previous error
Build completed unsuccessfully in 0:00:03

LLD cannot link correctly because the paths to object files that it sees are corrupted.

Removing [rust] use-lld = true from config.toml switches the linker to gnu and fixes the issue.
This is a relatively recent regression (maybe in Rust, maybe in LLD package shipped by MSYS2), things worked ok 3-4 months ago.

The text was updated successfully, but these errors were encountered:

petrochenkov · 2025-02-23T21:29:06Z

I didn't investigate this in detail yet, but I'll look during the week.
cc @mati865

mati865 · 2025-02-23T21:37:58Z

I'd expect this to be a Rust regression, barely 3 days ago MSYS2 built Rust 1.85 with LLD for CLANG* environments.

jieyouxu · 2025-02-24T08:18:07Z

Huh, is this related to the issue next doors #137499

ChrisDenton · 2025-02-24T11:23:25Z

That's unrelated as the error is in std. This looks more like a bootstrap or rustc issue. My guess would be something changed with how bootstrap handles paths? Or maybe something in cc-rs changed (which calls cc)? I'm not sure.

ognevny · 2025-02-25T08:47:31Z

I'd expect this to be a Rust regression, barely 3 days ago MSYS2 built Rust 1.85 with LLD for CLANG* environments.

note that I tried to build 1.84.0 with LLD for all envs some time ago, but it failed for non-clang environments: msys2/MINGW-packages#23350 (comment) (it was with use-lld = external though)

petrochenkov · 2025-02-25T15:35:55Z

The issue was caused by some change in Rust repo, commit d117b7f from Dec 31 works ok, but commit 854f225 from Jan 31 is failing. I continue bisecting.

petrochenkov · 2025-02-25T16:12:01Z

"Allow using self-contained LLD in bootstrap" did it.
cc @Kobzol

The question is why -Clink-arg=-fuse-ld=lld works when -Clinker-flavor=gnu-lld-cc does not.
I cannot reproduce the difference on simple rustc calls outside of bootstrap so far.

Kobzol · 2025-02-25T16:34:44Z

Feels like history repeats itself, last time I tried to switch to the new linker flavors in bootstrap, it also broke on Windows, IIRC 😆

I don't really develop on Windows, so it's a bit tricky for me to debug this. Maybe as a workaround I could use the new flavors in bootstrap only on Linux?

petrochenkov · 2025-02-25T19:00:08Z

The question is why -Clink-arg=-fuse-ld=lld works when -Clinker-flavor=gnu-lld-cc does not.

The difference is in the flavor of "response files" generated when the command line is too long.

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Line 1730 in c51b9b6

is_like_msvc: sess.target.is_like_msvc || (cfg!(windows) && flavor.uses_lld()),

-Clinker-flavor=gnu-lld-cc enables Windows-style escaping, which according to LLD source code (https://github.com/llvm/llvm-project/blob/3968ebd00da80a08de84f83a101ebb23710f6631/lld/COFF/DriverUtils.cpp#L768-L769) should work, but it doesn't actually work and produces errors from the log above.
Posix-style escaping in the same place works well, and everything builds.

cc #122596 @rcxdude

UPD: LLD for mingw has its own getQuotingStyle logic https://github.com/llvm/llvm-project/blob/main/lld/MinGW/Driver.cpp#L110-L114 but it also seems to produce Windows-style escaping.

petrochenkov · 2025-02-25T23:25:30Z

which according to LLD source code

I looked at wrong source code 🤦
The response file first goes to C compiler of course, not directly to LLD, and C compiler always uses Posix escaping.
I'll send the fix tomorrow.

rcxdude · 2025-02-25T23:31:23Z

The only thing I'd mention here is that the breakage I fixed only appears if you have spaces (maybe some other special characters?) in the path: Passing POSIX escaping to something expecting 'windows' escaping will mostly work except with spaces(because you can't escape them with backslashes), while the reverse will break more obviously(because backslashes get interpreted as escapes): I don't know if there's any test cases currently that will exercise the combination of a response file and spaces in the path. And at least on the configuration I was working with (MSVC isntead of mingw) it looked like LLD was always using MSVC-style.

petrochenkov · 2025-02-26T14:46:52Z

Fixed in #137676.

linker: Fix escaping style for response files on Windows If we use a С/С++ compiler as linker, then Posix-style escaping should be used. Also temporarily fixup rustbuild to not fail at least in common scenarios, until the bootstrap compiler is updated. Fixes rust-lang#137498

Rollup merge of rust-lang#137676 - petrochenkov:winresp, r=Kobzol linker: Fix escaping style for response files on Windows If we use a С/С++ compiler as linker, then Posix-style escaping should be used. Also temporarily fixup rustbuild to not fail at least in common scenarios, until the bootstrap compiler is updated. Fixes rust-lang#137498

petrochenkov added C-bug Category: This is a bug. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Feb 23, 2025

rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Feb 23, 2025

petrochenkov added O-windows-gnu Toolchain: GNU, Operating system: Windows needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Feb 23, 2025

jieyouxu added the E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc label Feb 23, 2025

saethlin removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Feb 24, 2025

jieyouxu added S-has-bisection Status: a bisection has been found for this issue A-linkers Area: linkers... you gotta love linkers and removed E-needs-bisection Call for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc labels Feb 25, 2025

petrochenkov mentioned this issue Feb 26, 2025

linker: Fix escaping style for response files on Windows #137676

Merged

bors closed this as completed in f1cdd3b Mar 1, 2025

petrochenkov mentioned this issue Mar 16, 2025

rust-lld on NixOS can't link the program if the target dir lives on a path with spaces in it #138513

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bootstrap: Build with `use-lld = true` fails under MSYS2 due to some path escaping issues #137498

bootstrap: Build with `use-lld = true` fails under MSYS2 due to some path escaping issues #137498

petrochenkov commented Feb 23, 2025

petrochenkov commented Feb 23, 2025

mati865 commented Feb 23, 2025

jieyouxu commented Feb 24, 2025

ChrisDenton commented Feb 24, 2025 •

edited

Loading

ognevny commented Feb 25, 2025 •

edited

Loading

petrochenkov commented Feb 25, 2025

petrochenkov commented Feb 25, 2025 •

edited

Loading

Kobzol commented Feb 25, 2025

petrochenkov commented Feb 25, 2025 •

edited

Loading

petrochenkov commented Feb 25, 2025

rcxdude commented Feb 25, 2025 •

edited

Loading

petrochenkov commented Feb 26, 2025

bootstrap: Build with use-lld = true fails under MSYS2 due to some path escaping issues #137498

bootstrap: Build with use-lld = true fails under MSYS2 due to some path escaping issues #137498

Comments

petrochenkov commented Feb 23, 2025

petrochenkov commented Feb 23, 2025

mati865 commented Feb 23, 2025

jieyouxu commented Feb 24, 2025

ChrisDenton commented Feb 24, 2025 • edited Loading

ognevny commented Feb 25, 2025 • edited Loading

petrochenkov commented Feb 25, 2025

petrochenkov commented Feb 25, 2025 • edited Loading

Kobzol commented Feb 25, 2025

petrochenkov commented Feb 25, 2025 • edited Loading

petrochenkov commented Feb 25, 2025

rcxdude commented Feb 25, 2025 • edited Loading

petrochenkov commented Feb 26, 2025

bootstrap: Build with `use-lld = true` fails under MSYS2 due to some path escaping issues #137498

bootstrap: Build with `use-lld = true` fails under MSYS2 due to some path escaping issues #137498

ChrisDenton commented Feb 24, 2025 •

edited

Loading

ognevny commented Feb 25, 2025 •

edited

Loading

petrochenkov commented Feb 25, 2025 •

edited

Loading

petrochenkov commented Feb 25, 2025 •

edited

Loading

rcxdude commented Feb 25, 2025 •

edited

Loading