Ensure crawler can't run out of space with --diskUtilization param #264
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ikreymer
reviewed
Mar 31, 2023
| @@ -148,6 +150,21 @@ export async function getDirSize(dir) { | |||
| return size; | |||
| } | |||
|
|
|||
| export async function getDiskUsage(path="/") { | |||
| const exec = util.promisify(child_process.exec); | |||
| const result = await exec(`df ${path}`); | |||
There was a problem hiding this comment.
I wonder if this is the best way to check disk usage... i guess maybe it is, instead of using some node library for it?
I suppose if it works well, can go with this for now..
There was a problem hiding this comment.
I looked for node libraries first (like https://www.npmjs.com/package/check-disk-space and https://www.npmjs.com/package/diskusage) but they all seemed to be wrappers for OS tools like df or its Windows equivalent so I figured I'd spare us another dependency and just implement it myself.
ikreymer
reviewed
Mar 31, 2023
| const kbUsed = parseInt(diskUsage["Used"]); | ||
| const kbTotal = parseInt(diskUsage["1K-blocks"]); | ||
| let kbArchiveDirSize = Math.floor(size/1024); | ||
| if (this.params.combineWARC && this.params.generateWACZ) { |
There was a problem hiding this comment.
Nice, yes, this is a better way to do this!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #242
The first commit implements
--diskUtilizationas described in the issue. I found that this stopped a lot of crawls prematurely in local testing even though the total size, even after combining WARCs and generating WACZ, still wouldn't have crossed the disk threshold.The second commit modifies the routine to keep the disk utilization percentage high (90% by default) but instead projects whether that threshold is likely to be reached based on the size of the current archive directory (x4 if combineWARC and generateWACZ are passed, x2 if only one of them).