Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

False positives reported with --incompatible_disallow_empty_glob flag #8517

Closed
davido opened this issue May 30, 2019 · 5 comments
Closed

False positives reported with --incompatible_disallow_empty_glob flag #8517

davido opened this issue May 30, 2019 · 5 comments
Assignees
Labels
P1 I'll work on this now. (Assignee required) type: bug

Comments

@davido
Copy link
Contributor

davido commented May 30, 2019

On Bazel@HEAD (b2fac74) I'm seeing false positive reported on Gerrit Code Review repository (stable-2.14 branch):

$ b10 build --incompatible_disallow_empty_glob=true plugins/replication
INFO: Invocation ID: 72331e2c-6ce4-4253-9f8b-78ce226e0e77
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_toolchains/rules/version_check.bzl:45:9: 
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '0.25.2'
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_toolchains/rules/version_check.bzl:45:9: 
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '0.25.2'
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:96:13: Current Bazel is not a release version; cannot check for compatibility. Make sure that you are running at least Bazel 0.25.0.
ERROR: /home/davido/projects/gerrit2/plugins/replication/BUILD:6:12: Traceback (most recent call last):
	File "/home/davido/projects/gerrit2/plugins/replication/BUILD", line 4
		gerrit_plugin(name = "replication", srcs = glob(..."]), <3 more arguments>)
	File "/home/davido/projects/gerrit2/plugins/replication/BUILD", line 6, in gerrit_plugin
		glob(["src/main/java/**/*.java"])
all files in the glob have been excluded, but allow_empty is set to False.
ERROR: /home/davido/projects/gerrit2/plugins/replication/BUILD:23:12: Traceback (most recent call last):
	File "/home/davido/projects/gerrit2/plugins/replication/BUILD", line 21
		junit_tests(name = "replication_tests", srcs =..."]), <3 more arguments>)
	File "/home/davido/projects/gerrit2/plugins/replication/BUILD", line 23, in junit_tests
		glob(["src/test/java/**/*Test.java"])
all files in the glob have been excluded, but allow_empty is set to False.
ERROR: /home/davido/projects/gerrit2/plugins/replication/BUILD:37:12: Traceback (most recent call last):
	File "/home/davido/projects/gerrit2/plugins/replication/BUILD", line 34
		java_library(name = "replication_util", testonl..., <2 more arguments>)
	File "/home/davido/projects/gerrit2/plugins/replication/BUILD", line 37, in java_library
		glob(["src/test/java/**/*.java"], exclu..."])
all files in the glob have been excluded, but allow_empty is set to False.
ERROR: Skipping 'plugins/replication': no such target '//plugins/replication:replication': target 'replication' not declared in package 'plugins/replication' defined by /home/davido/projects/gerrit2/plugins/replication/BUILD
WARNING: Target pattern parsing failed.
ERROR: no such target '//plugins/replication:replication': target 'replication' not declared in package 'plugins/replication' defined by /home/davido/projects/gerrit2/plugins/replication/BUILD
INFO: Elapsed time: 0.083s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

This flag is tracked here.

@laurentlb
Copy link
Contributor

Thanks for the report.

Can you do print(glob(["src/main/java/**/*.java"])) and confirm it contains files (with --incompatible_disallow_empty_glob=false)?

@davido
Copy link
Contributor Author

davido commented May 31, 2019

I added print statement, but I cannot reproduce it any more, as I see reports in Bazel itself:

$ b10 build --incompatible_disallow_empty_glob=true plugins/replication
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 5c02393d-07ee-4019-958e-d37907bd2ac5
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_toolchains/rules/version_check.bzl:45:9: 
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '0.25.2'
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_toolchains/rules/version_check.bzl:45:9: 
Current running Bazel is not a release version and one was not defined explicitly in rbe_autoconfig target. Falling back to '0.25.2'
DEBUG: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_skylib/lib/versions.bzl:96:13: Current Bazel is not a release version; cannot check for compatibility. Make sure that you are running at least Bazel 0.25.0.
DEBUG: /home/davido/projects/gerrit2/plugins/replication/BUILD:6:12: ["src/main/java/com/googlesource/gerrit/plugins/replication/AutoReloadConfigDecorator.java", "src/main/java/com/googlesource/gerrit/plugins/replication/AutoReloadSecureCredentialsFactoryDecorator.java", "src/main/java/com/googlesource/gerrit/plugins/replication/CredentialsFactory.java", "src/main/java/com/googlesource/gerrit/plugins/replication/Destination.java", "src/main/java/com/googlesource/gerrit/plugins/replication/DestinationConfiguration.java", "src/main/java/com/googlesource/gerrit/plugins/replication/DestinationFactory.java", "src/main/java/com/googlesource/gerrit/plugins/replication/Init.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ListCommand.java", "src/main/java/com/googlesource/gerrit/plugins/replication/OnStartStop.java", "src/main/java/com/googlesource/gerrit/plugins/replication/PushAll.java", "src/main/java/com/googlesource/gerrit/plugins/replication/PushOne.java", "src/main/java/com/googlesource/gerrit/plugins/replication/PushResultProcessing.java", "src/main/java/com/googlesource/gerrit/plugins/replication/RefReplicatedEvent.java", "src/main/java/com/googlesource/gerrit/plugins/replication/RefReplicationDoneEvent.java", "src/main/java/com/googlesource/gerrit/plugins/replication/RemoteSiteUser.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationConfig.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationFileBasedConfig.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationFilter.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationLogFile.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationMetrics.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationModule.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationQueue.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationScheduledEvent.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationSshSessionFactoryProvider.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationState.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationStateListener.java", "src/main/java/com/googlesource/gerrit/plugins/replication/ReplicationStateLogger.java", "src/main/java/com/googlesource/gerrit/plugins/replication/SecureCredentialsFactory.java", "src/main/java/com/googlesource/gerrit/plugins/replication/SecureCredentialsProvider.java", "src/main/java/com/googlesource/gerrit/plugins/replication/SshModule.java", "src/main/java/com/googlesource/gerrit/plugins/replication/StartCommand.java", "src/main/java/com/googlesource/gerrit/plugins/replication/StartReplicationCapability.java"]
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:100:12: Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 97
		filegroup(name = "extdir", deprecation = DEP..., ..."]))
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 100, in filegroup
		glob(["jre/lib/ext/*.jar"])
glob pattern 'jre/lib/ext/*.jar' didn't match anything, but allow_empty is set to False.
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:106:12: Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 103
		filegroup(name = "extclasspath", deprecation..., ..."]))
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 106, in filegroup
		glob(["jre/lib/ext/*.jar"])
glob pattern 'jre/lib/ext/*.jar' didn't match anything, but allow_empty is set to False.
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:115:21: Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 109
		filegroup(name = "jre-bin", srcs = select({"..."])}), ...)
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 111, in filegroup
		select({":windows": glob(["jre/bin/**"]..."])})
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 115, in select
		glob(["jre/bin/**"], exclude = ["jre/bi..."])
glob pattern 'jre/bin/**' didn't match anything, but allow_empty is set to False.
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:123:12: Traceback (most recent call last):
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 121
		filegroup(name = "jre-lib", srcs = glob(["jr..."]))
	File "/home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel", line 123, in filegroup
		glob(["jre/lib/**"])
glob pattern 'jre/lib/**' didn't match anything, but allow_empty is set to False.
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:131:1: Target '@remotejdk11_linux//:jre-bin' contains an error and its package is in error and referenced by '@remotejdk11_linux//:jre-default'
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:131:1: Target '@remotejdk11_linux//:jre-lib' contains an error and its package is in error and referenced by '@remotejdk11_linux//:jre-default'
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:166:1: Target '@remotejdk11_linux//:jdk-bin' contains an error and its package is in error and referenced by '@remotejdk11_linux//:jdk'
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:166:1: Target '@remotejdk11_linux//:jdk-include' contains an error and its package is in error and referenced by '@remotejdk11_linux//:jdk'
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:166:1: Target '@remotejdk11_linux//:jdk-lib' contains an error and its package is in error and referenced by '@remotejdk11_linux//:jdk'
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/remotejdk11_linux/BUILD.bazel:166:1: Target '@remotejdk11_linux//:jre-default' contains an error and its package is in error and referenced by '@remotejdk11_linux//:jdk'
ERROR: /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/external/bazel_tools/tools/jdk/BUILD:492:1: Target '@remotejdk11_linux//:jdk' contains an error and its package is in error and referenced by '@bazel_tools//tools/jdk:remote_jdk11'
ERROR: Analysis of target '//plugins/replication:replication' failed; build aborted: Analysis failed
INFO: Elapsed time: 55.523s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (76 packages loaded, 1627 targets configured)
    Fetching @user; fetching 46s
    Fetching @dev; fetching 46s
    Fetching @lucene-core; fetching 39s
    Fetching @jgit-lib; fetching 22s
    Fetching @guava; fetching 20s
    Fetching @soy; fetching 9s
    Fetching @grappa; fetching

@laurentlb laurentlb added P1 I'll work on this now. (Assignee required) type: bug and removed untriaged labels May 31, 2019
@laurentlb
Copy link
Contributor

Thanks a lot for the report! I can reproduce the problem on Bazel source code itself.

Please ignore the flag incompatible_disallow_empty_glob in 0.27, we should fix it for 0.28.

@laurentlb
Copy link
Contributor

There's a correctness issue.
I tried with this BUILD file:

print(glob(["*"], allow_empty = False))

It correctly reports the files from the current package. Then I modify the file (e.g. add a comment), and I get an error:

		glob(["*"], allow_empty = False)
all files in the glob have been excluded, but allow_empty is set to False.

Looking at Bazel code, I see the error is raised by globUnsorted. In the 2nd build, globUnsorted is called with an empty list of includes. I don't know why.

If I add the naive workaround:

    if (includes.isEmpty()) { return new ArrayList<>(); }

Everything seems to work fine.

cc @haxorz

@haxorz
Copy link
Contributor

haxorz commented Jun 12, 2019

cursory analysis (i merely skimmed the source code and didn't e.g. repro in a debugger):

  • on the 2nd build, we get skyframe cache hits for the GlobValue deps. so that's why PackageFunction.SkyframeHybridGlobber#runAsync's 'globsToDelegate' is empty.
  • we should have implemented the 'allow_empty' feature in PackageFunction.SkyframeHybridGlobber, not in GlobCache. i'm sorry for not noticing this during code review of commit b340fb0.

@haxorz haxorz self-assigned this Jun 12, 2019
irengrig pushed a commit to irengrig/bazel that referenced this issue Jun 18, 2019
…ow_empty' param to the 'glob' function

(1) Previously, the 'allow_empty=False' enforcement was solely in legacy globbing. So on incremental package evaluation where all the GlobValue nodes already existed in the Skyframe graph, we would not do any enforcement. This was fixed by having by Skyframe globbing and legacy globbing do enforcement.

(2) On incremental package evaluation where all the GlobValue nodes already existed in the Skyframe graph, we would still call into legacy globbing with an empty list of glob patterns. Legacy globbing would then error out just as it does normally when a glob *include* pattern matches a bunch of stuff, but then all the matches are removed by an *exclude* pattern. This was fixed by having Skyframe hybrid globbing only call into legacy globbing if there is work to do.

I added a bunch of unit tests for various incremental situations. I also added explicit unit tests for 'allow_empty=True'. We were previous missing all/some of this coverage, respectively.

Fixes bazelbuild#8517

RELNOTES: None
PiperOrigin-RevId: 253046516
siberex pushed a commit to siberex/bazel that referenced this issue Jul 4, 2019
…ow_empty' param to the 'glob' function

(1) Previously, the 'allow_empty=False' enforcement was solely in legacy globbing. So on incremental package evaluation where all the GlobValue nodes already existed in the Skyframe graph, we would not do any enforcement. This was fixed by having by Skyframe globbing and legacy globbing do enforcement.

(2) On incremental package evaluation where all the GlobValue nodes already existed in the Skyframe graph, we would still call into legacy globbing with an empty list of glob patterns. Legacy globbing would then error out just as it does normally when a glob *include* pattern matches a bunch of stuff, but then all the matches are removed by an *exclude* pattern. This was fixed by having Skyframe hybrid globbing only call into legacy globbing if there is work to do.

I added a bunch of unit tests for various incremental situations. I also added explicit unit tests for 'allow_empty=True'. We were previous missing all/some of this coverage, respectively.

Fixes bazelbuild#8517

RELNOTES: None
PiperOrigin-RevId: 253046516
irengrig pushed a commit to irengrig/bazel that referenced this issue Jul 15, 2019
…ow_empty' param to the 'glob' function

(1) Previously, the 'allow_empty=False' enforcement was solely in legacy globbing. So on incremental package evaluation where all the GlobValue nodes already existed in the Skyframe graph, we would not do any enforcement. This was fixed by having by Skyframe globbing and legacy globbing do enforcement.

(2) On incremental package evaluation where all the GlobValue nodes already existed in the Skyframe graph, we would still call into legacy globbing with an empty list of glob patterns. Legacy globbing would then error out just as it does normally when a glob *include* pattern matches a bunch of stuff, but then all the matches are removed by an *exclude* pattern. This was fixed by having Skyframe hybrid globbing only call into legacy globbing if there is work to do.

I added a bunch of unit tests for various incremental situations. I also added explicit unit tests for 'allow_empty=True'. We were previous missing all/some of this coverage, respectively.

Fixes bazelbuild#8517

RELNOTES: None
PiperOrigin-RevId: 253046516
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 I'll work on this now. (Assignee required) type: bug
Projects
None yet
Development

No branches or pull requests

4 participants