Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify ncpu/workers/processes for each parallelisation step #750

Closed
ddobie opened this issue Sep 18, 2024 · 1 comment
Closed

Specify ncpu/workers/processes for each parallelisation step #750

ddobie opened this issue Sep 18, 2024 · 1 comment
Assignees

Comments

@ddobie
Copy link
Contributor

ddobie commented Sep 18, 2024

Currently the pipeline uses the one less than the number of CPUs available for all parallel operations.

This is silly because

  1. the user should be able to specify how many are used in general (e.g. when running on a shared machine)
  2. it is not necessarily efficient to base the number of workers on the number of cores available.

For example, the forced fitting parallelisation is limited foremost by IO (on nimbus using >~ 10 workers slows things down), but also by memory (each worker loads an image, background map and rms map or ~2.1GB of data in total, into RAM)

We should therefore add arguments in the config file for the user to specify how many workers should be used for each parallel computation.

@ddobie
Copy link
Contributor Author

ddobie commented Nov 13, 2024

Closed by #777

@ddobie ddobie closed this as completed Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants