Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Experiments in trying to speed up CI #21010

Closed
wants to merge 9 commits into from

Conversation

itamarst
Copy link
Contributor

@itamarst itamarst commented Jan 30, 2025

  • sccache instead of normal all-or-nothing GHA cache
  • Give test python builds their own profile, since it seems like they were never getting cached at all. My guess is that custom RUSTFLAGS meant the code coverage build was trashing any relevant cache.

Copy link

codecov bot commented Jan 30, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.17%. Comparing base (4fc5929) to head (f97bd1d).
Report is 27 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #21010      +/-   ##
==========================================
- Coverage   79.20%   79.17%   -0.04%     
==========================================
  Files        1583     1583              
  Lines      225105   225255     +150     
  Branches     2581     2586       +5     
==========================================
+ Hits       178294   178337      +43     
- Misses      46221    46328     +107     
  Partials      590      590              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@itamarst itamarst changed the title Experiment: using sccache instead of normal Rust GHA cache perf: Experiments in trying to speed up CI Jan 31, 2025
@github-actions github-actions bot added performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars and removed title needs formatting labels Jan 31, 2025
@itamarst
Copy link
Contributor Author

Notes for when I return to this next week:

  • Unclear if sccache does anything.
  • Why do test-python jobs appear to recompile everything? Was in the middle of checking if maturin develop -m py-polars/Cargo.toml made a difference, that's what the coverage job does (that one doesn't recompile everything?!).
  • adding pytest -n auto to io tests was useful. Next step is forcing rebuild, ideally with no code changes.
  • why is uv on windows slow?
  • check if --verbose on uv pip install is slowing things down.
  • check if tests are sufficiently faster with higher optimization level to merit switching.

@itamarst itamarst closed this Feb 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance issues or improvements python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants