Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Cannot get node id for DirectoryArtifactValue" on tree artifact containing symlink to directory + remote cache + BES #20415

Closed
tjgq opened this issue Dec 2, 2023 · 3 comments
Closed

"Cannot get node id for DirectoryArtifactValue" on tree artifact containing symlink to directory + remote cache + BES #20415

tjgq opened this issue Dec 2, 2023 · 3 comments
Assignees
@tjgq
Labels
team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug untriaged

Comments

@tjgq
Copy link
Contributor

tjgq commented Dec 2, 2023

Repro:

defs.bzl

def _impl(ctx):
    d = ctx.actions.declare_directory(ctx.label.name)
    ctx.actions.run_shell(
        outputs = [d],
        command = "cd $1 && mkdir dir && touch dir/file.txt && ln -s dir sym",
        arguments = [d.path],
    )
    return DefaultInfo(files = depset([d]))

buggy = rule(_impl)

BUILD

load("//:defs.bzl", "buggy")

buggy(
    name = "buggy",
)

.bazelrc

build --remote_instance_name=projects/bazel-untrusted/instances/default_instance
build --remote_cache=grpcs://remotebuildexecution.googleapis.com
build --remote_download_toplevel
build --google_default_credentials
build --bes_backend=buildeventservice.googleapis.com
build --bes_instance_name=bazel-untrusted
build --bes_results_url=https://source.cloud.google.com/results/invocations

Build once to populate the remote cache, then clean and rebuild. The following stack trace is produced:

java.lang.RuntimeException: Unexpected Exception 'Cannot get node id for DirectoryArtifactValue{mtime=1701528378671}' when closing BEP transports, this is a bug.
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceModule.waitForBuildEventTransportsToClose(BuildEventServiceModule.java:543)
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceModule.closeBepTransports(BuildEventServiceModule.java:625)
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceModule.afterCommand(BuildEventServiceModule.java:640)
	at com.google.devtools.build.lib.runtime.BlazeRuntime.afterCommand(BlazeRuntime.java:627)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:688)
	at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:244)
	at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
	at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:621)
	at io.grpc.Context$1.run(Context.java:566)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: Cannot get node id for DirectoryArtifactValue{mtime=1701528378671}
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:571)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:111)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:247)
	at com.google.devtools.build.lib.buildeventservice.BuildEventServiceModule.waitForBuildEventTransportsToClose(BuildEventServiceModule.java:529)
	... 11 more
Caused by: java.lang.UnsupportedOperationException: Cannot get node id for DirectoryArtifactValue{mtime=1701528378671}
	at com.google.devtools.build.lib.remote.RemoteActionFileSystem$2.getNodeId(RemoteActionFileSystem.java:665)
	at com.google.devtools.build.lib.vfs.DigestUtils$CacheKey.<init>(DigestUtils.java:67)
	at com.google.devtools.build.lib.vfs.DigestUtils.manuallyComputeDigest(DigestUtils.java:193)
	at com.google.devtools.build.lib.vfs.DigestUtils.getDigestWithManualFallback(DigestUtils.java:160)
	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:72)
	at com.google.devtools.build.lib.remote.util.DigestUtil.compute(DigestUtil.java:67)
	at com.google.devtools.build.lib.remote.ByteStreamBuildEventArtifactUploader.readPathMetadata(ByteStreamBuildEventArtifactUploader.java:239)
	at com.google.devtools.build.lib.remote.ByteStreamBuildEventArtifactUploader.lambda$doUpload$8(ByteStreamBuildEventArtifactUploader.java:413)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableMap$MapSubscriber.onNext(FlowableMap.java:64)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.fastPath(FlowableFromIterable.java:185)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:129)
	at io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153)
	at io.reactivex.rxjava3.internal.jdk8.FlowableCollectWithCollectorSingle$CollectorSingleObserver.onSubscribe(FlowableCollectWithCollectorSingle.java:102)
	at io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15917)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15917)
	at io.reactivex.rxjava3.internal.jdk8.FlowableCollectWithCollectorSingle.subscribeActual(FlowableCollectWithCollectorSingle.java:71)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap.subscribeActual(SingleFlatMap.java:37)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.single.SingleUsing.subscribeActual(SingleUsing.java:83)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
	at io.reactivex.rxjava3.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
	at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:38)
	at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:25)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	... 3 more

Reproducible on 7.0.0rc5. On 6.4.0 it fails due to a different error, which can be fixed with #20409. See original report at #20408.
@tjgq tjgq mentioned this issue Dec 2, 2023
Closed
@tjgq tjgq self-assigned this Dec 2, 2023
@tjgq tjgq added team-Remote-Exec Issues and PRs for the Execution (Remote) team untriaged type: bug labels Dec 2, 2023
@tjgq tjgq changed the title "Cannot get node id for DirectoryArtifactValue" on tree artifact containing symlink + remote cache + BES "Cannot get node id for DirectoryArtifactValue" on tree artifact containing symlink to directory + remote cache + BES Dec 2, 2023
@brentleyjones
Copy link
Contributor

@bazel-io flag

@bazel-io bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Dec 2, 2023
@keertk
Copy link
Member

keertk commented Dec 2, 2023

@bazel-io fork 7.0.0

@bazel-io bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label Dec 2, 2023
@bazel-io bazel-io mentioned this issue Dec 2, 2023
Closed
This was referenced Dec 3, 2023
Closed
Closed
tjgq added a commit to tjgq/bazel that referenced this issue Dec 3, 2023
@tjgq
Fix LocalFileType for symlink to directory inside tree artifact.
1e44dcb 
When a tree artifact contains a symlink to a directory, the TreeArtifactValue
contains a TreeFileArtifact mapping to a DirectoryArtifactValue (see bazelbuild#20418).

Because the file type to be uploaded to the BES is determined solely by the
Artifact type, this causes the uploader to attempt to upload a directory as if
it were a file. This fails silently when building with the bytes; when building
without the bytes, it crashes when attempting to digest the (in-memory)
directory, which has no associated inode (see bazelbuild#20415).

This PR fixes the file type computation to take into account both the artifact
and its metadata, preferring the latter when available.

Fixes bazelbuild#20415.
tjgq added a commit to tjgq/bazel that referenced this issue Dec 3, 2023
@tjgq
Fix LocalFileType computation when the artifact and its metadata disa…
e50eb0c 
…gree.

When a tree artifact contains a symlink to a directory, the TreeArtifactValue
contains a TreeFileArtifact mapping to a DirectoryArtifactValue (see bazelbuild#20418).

Because the file type to be uploaded to the BES is determined solely by the
Artifact type, this causes the uploader to attempt to upload a directory as if
it were a file. This fails silently when building with the bytes; when building
without the bytes, it crashes when attempting to digest the (in-memory)
directory, which has no associated inode (see bazelbuild#20415).

This PR fixes the file type computation to take into account both the artifact
and its metadata, preferring the latter when available.

Fixes bazelbuild#20415.
tjgq added a commit to tjgq/bazel that referenced this issue Dec 3, 2023
@tjgq
Fix LocalFileType computation when the artifact and its metadata disa…
86088e8 
…gree.

When a tree artifact contains a symlink to a directory, the TreeArtifactValue
contains a TreeFileArtifact mapping to a DirectoryArtifactValue (see bazelbuild#20418).

Because the file type to be uploaded to the BES is determined solely by the
Artifact type, this causes the uploader to attempt to upload a directory as if
it were a file. This fails silently when building with the bytes; when building
without the bytes, it crashes when attempting to digest the (in-memory)
directory, which has no associated inode (see bazelbuild#20415).

This PR fixes the file type computation to take into account both the artifact
and its metadata, preferring the latter when available.

Fixes bazelbuild#20415.
tjgq added a commit to tjgq/bazel that referenced this issue Dec 3, 2023
@tjgq
Fix LocalFileType computation when the artifact and its metadata disa…
370768c 
…gree.

When a tree artifact contains a symlink to a directory, the TreeArtifactValue
contains a TreeFileArtifact mapping to a DirectoryArtifactValue (see bazelbuild#20418).

Because the file type to be uploaded to the BES is determined solely by the
Artifact type, this causes the uploader to attempt to upload a directory as if
it were a file. This fails silently when building with the bytes; when building
without the bytes, it crashes when attempting to digest the (in-memory)
directory, which has no associated inode (see bazelbuild#20415).

This PR fixes the file type computation to take into account both the artifact
and its metadata, preferring the latter when available.

Fixes bazelbuild#20415.
tjgq added a commit to tjgq/bazel that referenced this issue Dec 3, 2023
@tjgq
Fix LocalFileType computation when the artifact and its metadata disa…
17f085d 
…gree.

When a tree artifact contains a symlink to a directory, the TreeArtifactValue
contains a TreeFileArtifact mapping to a DirectoryArtifactValue (see bazelbuild#20418).

Because the file type to be uploaded to the BES is determined solely by the
Artifact type, this causes the uploader to attempt to upload a directory as if
it were a file. This fails silently when building with the bytes; when building
without the bytes, it crashes when attempting to digest the (in-memory)
directory, which has no associated inode (see bazelbuild#20415).

This PR fixes the file type computation to take into account both the artifact
and its metadata, preferring the latter when available.

Fixes bazelbuild#20415.
bazel-io pushed a commit to bazel-io/bazel that referenced this issue Dec 4, 2023
@tjgq @bazel-io
Fix LocalFileType computation when the artifact and its metadata disa…
a096f96 
…gree.

When a tree artifact contains a symlink to a directory, the TreeArtifactValue contains a TreeFileArtifact mapping to a DirectoryArtifactValue (see bazelbuild#20418).

Because the file type to be uploaded to the BES is determined solely by the Artifact type, this causes the uploader to attempt to upload a directory as if it were a file. This fails silently when building with the bytes; when building without the bytes, it crashes when attempting to digest the (in-memory) directory, which has no associated inode (see bazelbuild#20415).

This PR fixes the file type computation to take into account both the artifact and its metadata, preferring the latter when available.

Fixes bazelbuild#20415.

Closes bazelbuild#20419.

PiperOrigin-RevId: 587654070
Change-Id: Ib62bbaaed62b00c12bcabdc2bc9bee57aee0bcef
@bazel-io bazel-io mentioned this issue Dec 4, 2023
Merged
keertk pushed a commit that referenced this issue Dec 4, 2023
@bazel-io @tjgq
[7.0.0] "Cannot get node id for DirectoryArtifactValue" on tree artif…
8e2dfcd 
…act containing symlink to directory + remote cache + BES (#20424)

When a tree artifact contains a symlink to a directory, the
TreeArtifactValue contains a TreeFileArtifact mapping to a
DirectoryArtifactValue (see #20418).

Because the file type to be uploaded to the BES is determined solely by
the Artifact type, this causes the uploader to attempt to upload a
directory as if it were a file. This fails silently when building with
the bytes; when building without the bytes, it crashes when attempting
to digest the (in-memory) directory, which has no associated inode (see
#20415).

This PR fixes the file type computation to take into account both the
artifact and its metadata, preferring the latter when available.

Fixes #20415.

Closes #20419.

Commit
bb5ff63

PiperOrigin-RevId: 587654070
Change-Id: Ib62bbaaed62b00c12bcabdc2bc9bee57aee0bcef

Co-authored-by: Tiago Quelhas <[email protected]>
@iancha1992
Copy link
Member

A fix for this issue has been included in Bazel 7.0.0 RC6. Please test out the release candidate and report any issues as soon as possible. Thanks!

@ljessendk ljessendk mentioned this issue Feb 1, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tjgq tjgq

Labels
team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug untriaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix LocalFileType computation when the artifact and its metadata disagree. tjgq/bazel
5 participants
@brentleyjones @tjgq @bazel-io @keertk @iancha1992