Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metrics for FUSE worker idle and total count #1264

Merged
merged 1 commit into from
Feb 11, 2025

Conversation

dannycjones
Copy link
Contributor

@dannycjones dannycjones commented Feb 11, 2025

Before this change, there was no visibility into how many FUSE worker threads within Mountpoint had been created nor any indication if they are all busy or not.

This change adds both a count for the number of FUSE worker threads that have been spawned, as well as a count measuring how many are currently considered idle.

With the current metric implementation, these are guages and are only emitted when the value is updated. They are emitted in the logs as follows:

2025-02-11T13:00:14.647185Z  INFO mountpoint_s3::metrics: fuse.mp_workers.idle_count: 3
2025-02-11T13:00:14.647227Z  INFO mountpoint_s3::metrics: fuse.mp_workers.total_count: 4
2025-02-11T13:00:19.659416Z  INFO mountpoint_s3::metrics: fuse.mp_workers.idle_count: 3
2025-02-11T13:00:24.672336Z  INFO mountpoint_s3::metrics: fuse.mp_workers.idle_count: 3
2025-02-11T13:00:29.685867Z  INFO mountpoint_s3::metrics: fuse.mp_workers.idle_count: 4

Does this change impact existing behavior?

This adds a new metric only.

Does this change need a changelog entry? Does it require a version change?

No, new metric only. Metrics aren't advertised as a stable feature.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

@dannycjones dannycjones requested a review from passaro February 11, 2025 13:07
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:07 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:07 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:07 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:07 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:07 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:07 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:07 — with GitHub Actions Inactive
Signed-off-by: Daniel Carl Jones <[email protected]>
@dannycjones dannycjones force-pushed the add-worker-count-metrics branch from 8141cfd to df4b665 Compare February 11, 2025 13:10
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:10 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:10 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:10 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:10 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:10 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:10 — with GitHub Actions Inactive
@dannycjones dannycjones temporarily deployed to PR integration tests February 11, 2025 13:10 — with GitHub Actions Inactive

let new_count = old_count + 1;
let idle_worker_count = self.state.idle_worker_count.fetch_add(1, Ordering::SeqCst) + 1;
metrics::gauge!(METRIC_NAME_FUSE_WORKERS_TOTAL).set(new_count as f64);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this also be set in start?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is set when start is called, since start will invoke this method.

Copy link
Contributor

@passaro passaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@dannycjones dannycjones added this pull request to the merge queue Feb 11, 2025
Merged via the queue into awslabs:main with commit e56d343 Feb 11, 2025
24 checks passed
@dannycjones dannycjones deleted the add-worker-count-metrics branch February 11, 2025 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants