Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(memory): Limit memory usage for restore when having large pack files #165

Merged
merged 1 commit into from
Feb 21, 2024
Merged

fix(memory): Limit memory usage for restore when having large pack files #165

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion crates/core/src/commands/restore.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ use crate::{
pub(crate) mod constants {
/// The maximum number of reader threads to use for restoring.
pub(crate) const MAX_READER_THREADS_NUM: usize = 20;
/// The maximum size of pack-part which is read at once from the backend.
/// (needed to limit the memory size used for large backends)
pub(crate) const LIMIT_PACK_READ: u32 = 40 * 1024 * 1024; // 40 MiB
}

type RestoreInfo = BTreeMap<(Id, BlobLocation), Vec<FileLocation>>;
Expand Down Expand Up @@ -461,8 +464,14 @@ fn restore_contents<P: ProgressBars, S: Open>(
.collect();
(pack, bl.offset, bl.length, from_file, name_dests)
})
// optimize reading from backend by reading many blobs in a row
.coalesce(|mut x, mut y| {
if x.0 == y.0 && x.3.is_none() && y.1 == x.1 + x.2 {
if x.0 == y.0 // if the pack is identical
&& x.3.is_none() // and we don't read from a present file
&& y.1 == x.1 + x.2 // and the blobs are contiguous
// and we don't trespass the limit
&& x.2 + y.2 < constants::LIMIT_PACK_READ
{
x.2 += y.2;
x.4.append(&mut y.4);
Ok(x)
Expand Down
Loading