ERROR FAILED: //tests/core/transition:hermeticity_test with Bazel@HEAD in Downstream CI

[sgowroji]

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3109#0188d690-a57f-42b0-891f-8b2986d9db52

Platform: Windows, MacOs, Ububtu

Logs :

FAIL: //tests/core/transition:hermeticity_test 

Steps:

git clone https://github.com/bazelbuild/rules_go.git 
git reset 9d5416bcc9d48da3735b73fb5540b1cdbd470b85 --hard
export USE_BAZEL_VERSION=211fd6c14bef2f656265b3d3eb98a476f02ebf88
bazel build //...

Reference issue, which is causing error : bazelbuild/bazel@cb55897

CC Green Team @ fweikert

[fmeum]

@sgowroji How certain are you that the given commit is the culprit? Was there some kind of bisection that I could look into the results of?

[meteorcloudy]

It got the result from https://buildkite.com/bazel/culprit-finder/builds/5736#0188d70d-c9dc-438e-8119-76650a30b54c, which seems to be the same issue reported in bazelbuild/bazel#18666

[fmeum]

@meteorcloudy I think that there may be two different issues at play that confused the culprit finder. The log linked in the issue has a different kind of error: It looks like the output of cquery --output=jsonproto changed to new-line delimited JSON.

[fmeum]

Maybe this is caused by bazelbuild/bazel@607d0f7?

[meteorcloudy]

Oh, you are probably right, I forgot to check the actual error in the downstream pipeline.

[meteorcloudy]

/cc @zhengwei143

[zhengwei143]

Sorry I just saw this, let me take a quick look.

[zhengwei143]

I've just rolled back the change in bazelbuild/bazel@dba9e43 which should fix this.

[sgowroji]

We are also seeing this test failiure in the latest CI https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3160#01894849-482e-4d46-9522-beab4a40754e

//tests/core/output_groups:compilation_outputs_test                      FAILED 

[zhengwei143]

That looks like a separate issue - this test failure has been failing even after the rollback in bazelbuild/bazel@dba9e43 (see for example https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3149#01892e86-2c8f-4b6b-ad39-fb36384d135d)

Btw, I've reimplemented this in bazelbuild/bazel@aa88357, and the previous failures seem to be fixed. So this issue should be closed.

[fmeum]

I submitted #3623 to fix this separate issue - it is trivial and unrelated.

[sgowroji]

Hi @fmeum, I see rules_go is again failing with hermeticity_test failure in downstream https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3202#0189a9b6-589a-41d7-9870-a7a09b3436d7. Should we reopen this present issue or raise a new one. Could you please have a look. Thanks! 

//tests/core/transition:hermeticity_test                                

[zhengwei143]

Looking at the error logs, this should be due to bazelbuild/bazel@c061e57
which should also be fixed by the rollback in bazelbuild/bazel@0c91725.

[fmeum]

Since this change may be rolled forward as an incompatible change in the future, I went ahead and updated the usage: #3638

@sgowroji Either that or the rollback should fix this for the next downstream run

[sgowroji]

Hi @fmeum, We found rules_go Downstream CI is failing with hermeticity_test.

[fmeum]

Looks like an instance of bazelbuild/bazel#19823. I will disable Bzlmod for it.

[sgowroji]

Another test is failing in rules_go https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3519#018c569e-024d-4130-84d6-95989988cd21
//tests/integration/gopackagesdriver:gopackagesdriver\_test               FAILED

[fmeum]

@sgowroji Both failures should be fixed by #3781.

[fmeum]

rules_go is passing in the latest downstream pipeline: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3526