You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the pipeline uses the one less than the number of CPUs available for all parallel operations.
This is silly because
the user should be able to specify how many are used in general (e.g. when running on a shared machine)
it is not necessarily efficient to base the number of workers on the number of cores available.
For example, the forced fitting parallelisation is limited foremost by IO (on nimbus using >~ 10 workers slows things down), but also by memory (each worker loads an image, background map and rms map or ~2.1GB of data in total, into RAM)
We should therefore add arguments in the config file for the user to specify how many workers should be used for each parallel computation.
The text was updated successfully, but these errors were encountered:
Currently the pipeline uses the one less than the number of CPUs available for all parallel operations.
This is silly because
For example, the forced fitting parallelisation is limited foremost by IO (on nimbus using >~ 10 workers slows things down), but also by memory (each worker loads an image, background map and rms map or ~2.1GB of data in total, into RAM)
We should therefore add arguments in the config file for the user to specify how many workers should be used for each parallel computation.
The text was updated successfully, but these errors were encountered: