Example fails on macOS Big Sur (11.0.1)

[liam-ly]

Describe the bug
rules_haskell fails on newest macOS because of sed issue. Other downstream issues may exist.

To Reproduce

curl https://haskell.build/start | sh
bazel build //...

Starting local Bazel server and connecting to it...
INFO: Repository rules_haskell_ghc_darwin_amd64 instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule _ghc_bindist defined at:
  /private/var/tmp/_bazel_liam/8a2700302396c489f271b3938aaa9254/external/rules_haskell/haskell/ghc_bindist.bzl:362:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_haskell_ghc_darwin_amd64':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_liam/8a2700302396c489f271b3938aaa9254/external/rules_haskell/haskell/ghc_bindist.bzl", line 292
		execute_or_fail_loudly(ctx, <1 more arguments>)
	File "/private/var/tmp/_bazel_liam/8a2700302396c489f271b3938aaa9254/external/rules_haskell/haskell/private/workspace_utils.bzl", line 18, in execute_or_fail_loudly
		fail(<1 more arguments>)
Command failed: ./patch_bins
sed: bin/ghc: in-place editing only works for regular files
sed: bin/ghc-pkg: in-place editing only works for regular files
sed: bin/ghci: in-place editing only works for regular files
sed: bin/haddock: in-place editing only works for regular files
sed: bin/runghc: in-place editing only works for regular files
sed: bin/runhaskell: in-place editing only works for regular files
ERROR: /Users/liam/Work/bazel_test/BUILD.bazel:15:26: //:base depends on @rules_haskell_ghc_darwin_amd64//:toolchain-impl in repository @rules_haskell_ghc_darwin_amd64 which failed to fetch. no such package '@rules_haskell_ghc_darwin_amd64//': Traceback (most recent call last):
	File "/private/var/tmp/_bazel_liam/8a2700302396c489f271b3938aaa9254/external/rules_haskell/haskell/ghc_bindist.bzl", line 292
		execute_or_fail_loudly(ctx, <1 more arguments>)
	File "/private/var/tmp/_bazel_liam/8a2700302396c489f271b3938aaa9254/external/rules_haskell/haskell/private/workspace_utils.bzl", line 18, in execute_or_fail_loudly
		fail(<1 more arguments>)
Command failed: ./patch_bins
sed: bin/ghc: in-place editing only works for regular files
sed: bin/ghc-pkg: in-place editing only works for regular files
sed: bin/ghci: in-place editing only works for regular files
sed: bin/haddock: in-place editing only works for regular files
sed: bin/runghc: in-place editing only works for regular files
sed: bin/runhaskell: in-place editing only works for regular files
ERROR: Analysis of target '//:base' failed; build aborted: Analysis failed
INFO: Elapsed time: 84.056s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (19 packages loaded, 27 targets configured)

Expected behavior
Example builds

Environment

• OS name + version: macOS Big Sur version 11.0.1
• Bazel version: bazel 3.3.1
• Version of the rules: v0.13

Additional context

• There are likely additional issues when using Big Sur.
• HEAD of master also has the same issue

- http_archive(
-     name="rules_haskell",
-     strip_prefix="rules_haskell-0.13",
-     urls=["https://github.com/tweag/rules_haskell/archive/v0.13.tar.gz"],
-     sha256 = "b4e2c00da9bc6668fa0404275fecfdb31beb700abdba0e029e74cacc388d94d6",
- )
+ load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
+ git_repository(
+     name="rules_haskell",
+     remote="[email protected]:tweag/rules_haskell.git",
+     commit="9a6b89928a4f6171743798b8d7792b54bd16c5da",
+     # shallow_since="1592537100 -0700",
+ )

[aherrmann]

Thanks for reporting this! Unfortunately, I don't have a Big Sur machine available to reproduce this issue. The relevant code in rules_haskell is here. You can inspect the state of the filesystem after the failed operation by looking around in the following directory

$ cd $(bazel info output_base)/external/rules_haskell_ghc_darwin_amd64

Something that stands out: The files on which the error is reported are, I think, all symbolic links. On Linux, I see that sed is replacing the symlinks with actual files. Maybe this is something that changed on Big Sur. Either, it's disallowed to change symlinks inplace, or the issue is that it's trying to change the same file twice, once through the symlink and once through it's proper name.

A possible workaround would be to hard code the list of files to avoid accessing the symlinks (the GHC version may need to be adjusted):

diff --git a/haskell/ghc_bindist.bzl b/haskell/ghc_bindist.bzl
index 6194cf3b..feca3a73 100644
--- a/haskell/ghc_bindist.bzl
+++ b/haskell/ghc_bindist.bzl
@@ -281,11 +281,11 @@ def _ghc_bindist_impl(ctx):
         execute_or_fail_loudly(ctx, ["./configure", "--prefix", bindist_dir.realpath])
         execute_or_fail_loudly(ctx, ["make", "install"])
         ctx.file("patch_bins", executable = True, content = r"""#!/usr/bin/env bash
-grep --files-with-matches --null {bindist_dir} bin/* | xargs -0 -n1 \
     sed -i.bak \
         -e '2i\
           DISTDIR="$( dirname "$(resolved="$0"; while tmp="$(readlink "$resolved")"; do resolved="$tmp"; done; echo "$resolved")" )/.."' \
-        -e 's:{bindist_dir}:$DISTDIR:'
+        -e 's:{bindist_dir}:$DISTDIR:' \
+        bin/{ghc,ghc-pkg,ghci,haddock,runghc}-8.8.3
 """.format(
             bindist_dir = bindist_dir.realpath,
         ))

Once we know what changed in Big Sur to introduce this issue we could then try to properly fix the code.

[liam-ly]

I see this pattern in the bin directory

lrwxr-xr-x  1 liam  wheel   13 Nov 15 06:58 ghc-pkg -> ghc-pkg-8.6.5
-rwxr-xr-x  1 liam  wheel  400 Nov 15 06:59 ghc-pkg-8.6.5
-rwxr-xr-x  1 liam  wheel  636 Nov 15 06:58 ghc-pkg-8.6.5.bak

Seems like inplace editing using sed on a symlink no longer works, which sort of makes sense.

% touch foo-8.6.5
% ln -s foo-8.6.5 foo
% sed 's/a/b/g' foo-8.6.5
% sed -i.bak 's/a/b/g' foo-8.6.5
% sed -i.bak 's/a/b/g' foo
% sed 's/a/b/g' foo
sed: foo: in-place editing only works for regular files

One of the few references I could find to this error message included some source code from freebsd and apple opensource:

• https://opensource.apple.com/source/text_cmds/text_cmds-81/sed/main.c
• http://web.mit.edu/freebsd/head/usr.bin/sed/main.c

I can try something like a find bin -type f ... command in the ghc_bindist.bzl file next time I can come back to this.

[sjoerdvisscher]

On Catalina that last line doesn't give me an error indeed.

[aherrmann]

@liam-ly that's a great idea. I've opened #1445 which applies this change.

[liam-ly]

I was testing this idea out as well and it gets me further along the build 👍 . I will validate #1445 (since it's slightly different from what I had).

Unfortunately, I was seeing more downstream issues, potentially unrelated to rules_haskell since they are in the zlib compilation part, that I was trying to fix via configuration. The TL;DR version is

external/zlib.dev/gzlib.c:252:9: error: implicit declaration of function 'lseek' is invalid in C99 [-Werror,-Wimplicit-function-declaration]

[aherrmann]

It looks like the default CC toolchain enables -Werror which makes libz compilation fail. Disabling the warning should avoid that issue. See the added copts in b033122.

[kdeding]

@aherrmann, what's exact command (or option) to have -Werror disabled? I tried the this:

bazel build --copt="-Wno-error=execute_or_fail_loudly" //...

Still got the same error in the end. Not sure what else to try.

I'm new to this and seeing the same issue when trying to build github/semantic. Having googled for quite some time without any luck. Much appreciated.

[aherrmann]

@kdeding Adding the line

     copts = ["-Wno-implicit-function-declaration"],

to the cc_library target as shown here should avoid that issue.