pgsc_calc not using many resources: how do I configure to use more RAM?

[Fiwx]

I've read this, https://pgsc-calc.readthedocs.io/en/latest/how-to/bigjob.html, but I am a bit confused. I see that the HPC setup has each step configured with its own resources. I am running pgsc_calc locally, with a lot of RAM and many scores. All steps are using a very small portion (e.g., usually under 8 GB) of the total available RAM. Do I need to specify RAM usage for individual steps? For example, for 32 GB computers, I have the custom .config file passed in as a parameter set to this:

params {
    max_memory                 = '31.GB'
    max_cpus                   = 4
    max_time                   = '240.h'
}

However, this does not seem to be sufficient for allowing each step to use more resources. The pipeline takes a long time to finish so I'm hoping that being able to use more RAM will speed it up.

[nebfield]

The max parameters provide a ceiling to the amount of resources a process can use. If you set a step to request 10 cores but --max_cpus is 2, then only 2 cpus will be allocated - it's a threshold. So you either need to set the resources for each step or change the resources allocated to the process labels (process_low, process_medium) as described in the docs.

However, adding more resources is mostly helpful to make sure that larger datasets (like UK Biobank) will produce results. It doesn't mean the workflow will complete faster. For example, the COMBINE_SCOREFILES process is currently a single threaded python application, so it can only use 1 CPU even if you allocate 4 CPUs to the process (although we plan to improve this in the future). There are a lot of different factors affecting the speed of the overall workflow.

There's a Nextflow execution report in results/pipeline_info/ that will show:

• How many resources you requested
• What each process actually used
• How efficient the resource allocation was (e.g. 1 core requested and 1 core used = 100% efficiency, 2 cores requested but only 1 core used = 50% efficiency)