Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change index on mod_shredder.shredded_job to improve performance #1968

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

eiffel777
Copy link
Contributor

The xdmod-slurm-helper script runs the following query to try and find the time of the most recent job ingested in order to set the start time for the sacct command. Changing the order of the columns for the end_time index on mod_shredder.shredded provides a performance improvement. On metrics-dev, the query went from 6-8 minutes for the ub-hpc resource to around 30 seconds.

SELECT
  DATE_FORMAT(
    FROM_UNIXTIME(MAX(end_time)),
    '%Y-%m-%d %H:%i:%s'
   ) AS max_datetime
FROM shredded_job
WHERE resource_name = :resource

Tests performed

Tested in docker and on metrics-dev

Checklist:

  • The pull request description is suitable for a Changelog entry
  • The milestone is set correctly on the pull request
  • The appropriate labels have been added to the pull request

@eiffel777 eiffel777 added enhancement Enhancement of the functionality of an existing feature Category:General General labels Jan 30, 2025
@eiffel777 eiffel777 added this to the 11.5.0 milestone Jan 30, 2025
@eiffel777 eiffel777 self-assigned this Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category:General General enhancement Enhancement of the functionality of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant