Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lazy derive preprocessor function #2451

Open
6 tasks
bouweandela opened this issue Jun 10, 2024 · 3 comments
Open
6 tasks

Lazy derive preprocessor function #2451

bouweandela opened this issue Jun 10, 2024 · 3 comments
Labels
dask related to improvements using Dask preprocessor Related to the preprocessor

Comments

@bouweandela
Copy link
Member

bouweandela commented Jun 10, 2024

The derive preprocessor function does not (fully) support lazy derivation of the following variables

  • toz - used in recipe_perfmetrics_CMIP5.yml, recipe_eyring06jgr.yml
  • xco2 - used in recipe_gier2020bg.yml
  • xch4 - used in recipe_mpqb_xch4.yml
  • co2s - used in schlund20jgr/recipe_*.yml, recipe_gier2020bg.yml, recipe_wenzel16nat.yml
  • uajet - used in recipe_wenzel16jclim.yml
  • vegfrac - not used in any recipe Lazy derivation of vegfrac #2452

All of these derivation functions can also be 'tested' using examples/recipe_preprocessor_derive_test.yml.

Previous discussion on this topic in #42.

@bouweandela bouweandela added preprocessor Related to the preprocessor dask related to improvements using Dask labels Jun 10, 2024
@bouweandela
Copy link
Member Author

@chrisbillowsMO This would be a good issue to pick up if you're interested in doing Dask related work. While these derivation functions may seem a bit niche, they are used by some of the more computationally heavy recipes and they prevent is from running those with the distributed scheduler.

CC @ehogan

@chrisbillowsMO
Copy link
Contributor

@chrisbillowsMO This would be a good issue to pick up if you're interested in doing Dask related work. While these derivation functions may seem a bit niche, they are used by some of the more computationally heavy recipes and they prevent is from running those with the distributed scheduler.

CC @ehogan

Thanks for thinking of me @bouweandela! I will take a look as soon as I get a chance.

@bouweandela
Copy link
Member Author

Since some of these preprocessor functions predate the requirement of having unit tests, we usually test them manually by running examples/recipe_preprocessor_derive_test.yml and checking if the output is identical before and after we make changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dask related to improvements using Dask preprocessor Related to the preprocessor
Projects
Development

No branches or pull requests

2 participants