Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase JOBFS for configurations #83

Closed
blimlim opened this issue Aug 29, 2024 · 2 comments
Closed

Increase JOBFS for configurations #83

blimlim opened this issue Aug 29, 2024 · 2 comments

Comments

@blimlim
Copy link

blimlim commented Aug 29, 2024

I came across a jobfs quota exceeded error when running the historical+concentrations configuration for several years. It's not completely clear when in the run it happened, but I think it might have been during the setup stage:

Setting up ice
Setting up coupler
Checking exe, input and restart manifests
Job 123620242.gadi-pbs killed due to exceeding jobfs quota. Quota: 100.0MB, Used: 435.49MB, Host: gadi-cpu-clx-2284

At the bottom of the resource report it also has

 JobFS requested:    800.0MB                JobFS used: 672.63MB    

Rerunning got past the issue, but it could be good to increase the jobfs requested to try and avoid the error happening during long simulations.

@aidanheerdegen
Copy link
Member

Note that jobfs is shared over nodes

https://opus.nci.org.au/display/Help/PBS+Directives+Explained#PBSDirectivesExplained--ljobfs=%3C10GB%3E

On 48 cpu nodes ACCESS-ESM1.5 uses 8 nodes. If the JOBFS is used in setup it will only be the root node, so the usage will be concentrated on a single node.

Dale wrote a very nice explainer of JOBSFS in case anyone is interested

https://climate-cms.org/posts/2022-11-10-jobfs.html

(put this in a comment on the PR, but meant to chuck it here)

@blimlim
Copy link
Author

blimlim commented Aug 29, 2024

The explainer is really helpful! Closing this issue now that #84 and #85 are merged.

@blimlim blimlim closed this as completed Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants