Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drain: use authenticated ID as source of drained-by metadata #20317

Merged
merged 2 commits into from
Apr 9, 2024

Conversation

tgross
Copy link
Member

@tgross tgross commented Apr 8, 2024

When a node is set to drain, the state store reads the auth token off the request to record LastDrain metadata about the token used to drain the node. This code path in the state store can't correctly handle signed Workload Identity tokens or bearer tokens that may have expired (for example, while restarting a server and applying uncompacted Raft logs).

Rather than re-authenticating the request at the time of FSM apply, record the string derived from the authenticated identity as part of the Raft log entry.

Fixes: #17471

When a node is set to drain, the state store reads the auth token off the
request to record `LastDrain` metadata about the token used to drain the
node. This code path in the state store can't correctly handle signed Workload
Identity tokens or bearer tokens that may have expired (for example, while
restarting a server and applying uncompacted Raft logs).

Rather than re-authenticating the request at the time of FSM apply, record the
string derived from the authenticated identity as part of the Raft log
entry.

Fixes: #17471
nomad/structs/structs.go Outdated Show resolved Hide resolved
Copy link
Member

@jrasell jrasell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tgross tgross merged commit 2cf341b into main Apr 9, 2024
19 checks passed
@tgross tgross deleted the 17471-node-drain-acl branch April 9, 2024 13:28
philrenaud pushed a commit that referenced this pull request Apr 18, 2024
When a node is set to drain, the state store reads the auth token off the
request to record `LastDrain` metadata about the token used to drain the
node. This code path in the state store can't correctly handle signed Workload
Identity tokens or bearer tokens that may have expired (for example, while
restarting a server and applying uncompacted Raft logs).

Rather than re-authenticating the request at the time of FSM apply, record the
string derived from the authenticated identity as part of the Raft log
entry.

Fixes: #17471
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 13, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/1.5.x backport to 1.5.x release line backport/1.6.x backport to 1.6.x release line backport/1.7.x backport to 1.7.x release line theme/drain type/bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Auth error during node drain via Task API: rpc error: error looking up ACL token
3 participants