Option --path-separator failing on MSys (Git Bash)

[fawick]

I have fd 7.4.0 installed on a Windows machine (Rust 1.41.0, stable-x86_64-pc-windows-msvc) and run it from a bash that is provided by Git for Windows, which itself is based on Msys2 (according to https://github.com/git-for-windows/git/wiki/Technical-overview):

$ uname -a
MINGW64_NT-10.0-18363 SOMEHOST 3.0.7-338.x86_64 2019-07-03 15:16 UTC x86_64 Msys

$ fd
bar
bar\baz
bar\baz\bla
bar\baz\bla\some.txt
bar\yetanother.txt
other.txt

fd is using backslashes as it is supposed to do in Windows. #429 added the --path-separator option, which could be useful for using the results as paths within Git Bash. However, in my environment, the option fails, and every backslash is substituted with the absolute path to the Git for Windows installation directory:

$ fd --path-separator="/"
bar
barC:/Program Files/Git/baz
barC:/Program Files/Git/bazC:/Program Files/Git/bla
barC:/Program Files/Git/bazC:/Program Files/Git/blaC:/Program Files/Git/some.txt
barC:/Program Files/Git/yetanother.txt
other.txt

$ ls "C:/Program Files/Git/"
bin/  cmd/  dev/  etc/  git-bash.exe*  git-cmd.exe*  LICENSE.txt  mingw64/  ReleaseNotes.html  tmp/  unins000.dat  unins000.exe*  unins000.msg  usr/

Using single-quotes around the slash character make no difference. Nor does leaving out the quotes all together.

[sharkdp]

It looks to me like this is a problem of Msys2 or Git Bash. Could you please try these suggestions: https://stackoverflow.com/a/54924640/704831 ?

[fawick]

Thank you, that was a great hint, @sharkdp! I am very grateful to you for taking the time to dig into this.

For the record, one needs to set the environment variable MSYS_NO_PATHCONV=1:

$ fd --path-separator='/'
bar
barC:/Program Files/Git/baz
barC:/Program Files/Git/bazC:/Program Files/Git/bla
barC:/Program Files/Git/bazC:/Program Files/Git/blaC:/Program Files/Git/some.txt
barC:/Program Files/Git/yetanother.txt
other.txt

$ MSYS_NO_PATHCONV=1 fd --path-separator='/'
bar
bar/baz
bar/baz/bla
bar/baz/bla/some.txt
bar/yetanother.txt
other.txt

[chuxubank]

This MSYS_NO_PATHCONV=1 fix only works on Git Bash.
When comes to msys2, we should use MSYS2_ARG_CONV_EXCL=--path-separator
Ref: https://www.msys2.org/wiki/Porting/#filesystem-namespaces

[sharkdp]

Given that we introduced the --path-separator=… option for situations like Cygwin/MSYS2/Git Bash on Windows, it seems really unfortunate users have to go through this experience described in this ticket.

Maybe we should introduce a --path-separator-slash or --unix-path-separator option as an alias for --path-separator="/", which would not require any environment variables to be set?

What do you think?

[chuxubank]

Good idea!

[sharkdp]

Okay, let's turn this into a feature request.

[aswild]

I ran into this today trying to use fd on msys2, and just learned about MSYS2_ARG_CONV_EXCL so maybe I can fix my shell alias.

Could fd go a step further and try to auto-detect the path separator at runtime?

• MSYS2 sets the MSYSTEM environment variable, which could be checked cheaply
• Cygwin doesn't seem to set any unique environment variables
  • a hack could be to check for an entry starting with /cygdrive/ in PATH (usually PATH contains /cygdrive/c/Windows/System32, among other native windows path entries) Edit: apparently PATH gets mangled to a Windows-style string C:\foo\bar;C:\baz\quux at runtime, so that won't work.
  • I assume that forking off to a uname process would be too expensive
• I'm not sure about git bash, except that it's msys/mingw based. Does it also set MSYSTEM?

[sharkdp]

@aswild Great idea. Maybe this could help for cygwin: https://stackoverflow.com/a/3104453 ?

[aswild]

Ooh, OSTYPE looks so promising but apparently it's only set as a shell variable and isn't exported to the environment, foiled again.

[sharkdp]

It would be great if someone could summarize the current status and what should/could be done here to improve the situation.

[chuxubank]

I haven't used msys2 for a while. Instead, I'm using WSL now and no longer need this option.

I think it's OK to leave it unchanged since I already give the workaround in msys2 at #537 (comment)

[aswild]

It would be great if someone could summarize the current status and what should/could be done here to improve the situation.

My current view of the world:

• On MSYS (including all 3 flavors of MSYS2 shell and Git Bash), the MSYSTEM environment variable is exported, and programs can use that to detect an MSYS/MinGW shell. This is what I did on my fork in aswild@4441615
• You can globally disable MSYS' automatic path replacement (e.g. turning / into C:\msys64) by setting MSYS_NO_PATHCONV or MSYS2_ARG_CONV_EXCL, but you can also just write a double slash to disable it for a single argument, e.g. fd --path-separator=// ... (when fd executes, it will only see a single /)
• Cygwin is a mess on Rust in general
  • Cygwin is a unique environment that simultaneously both Unix and Windows. It works for C code because of the special Cygwin libc that hooks into all the low-level functions that would be syscalls on Linux.
  • Rust itself doesn't know about this (I see libstd contains some "raw" syscalls implemented in Rust rather than calling libc), and I think things go all the way back to missing LLVM support.
  • You can set the right environment variables to use rustup (x86_64-pc-windows-gnu) inside of Cygwin, but I've never been able to build anything that requires calling out to GCC or other external libraries, only pure-rust crates. I tried and failed attempting to get rustup to even find a MinGW version of GCC from within Cygwin.
• There's no sane way to detect Cygwin at runtime (without custom user-set environment variables) that I know of.
  • Cygwin doesn't do automatic path conversion like MSYS, because it expects that you'll be running Cygwin-compiled programs which understand its Unix-like filesystem internally (which native Win32 programs, including anything Rust, do not)
  • Thus, to use any Rust program from Cygwin, it must be compiled in a MinGW environment, and you need some sort of wrapper function to call cygpath -w on anything that looks like an absolute path. For example, here's what I did for fd: https://github.com/aswild/linuxfiles/blob/master/myshrc#L719
• WSL is great and gaining in popularity, so I bet that the Cygwin issues inherent to Rust may never get fixed - anyone deep enough in Rust internals to fix them probably has better things to spend their time on

What could fd do?

• @sharkdp if you like it, feel free to cherry-pick or adapt aswild@4441615 (or I can turn it into a PR). I think this fixes the common use case of fd in MSYS(2)/Git Bash and doesn't require users to learn about the PATHCONV/ARG_CONV environment variables or set shell aliases/functions.
• Acknowledge/document that Rust is inherently limited in Cygwin for reasons that Rust programs can't control or cleanly work around on their own.

[sharkdp]

@sharkdp if you like it, feel free to cherry-pick or adapt aswild@4441615 (or I can turn it into a PR). I think this fixes the common use case of fd in MSYS(2)/Git Bash and doesn't require users to learn about the PATHCONV/ARG_CONV environment variables or set shell aliases/functions.

A PR would be very much appreciated - thank you!