Performance Numbers ‐ v0.14

The procedure

The page cache is dropped between each tests.

Dataset are reduplicated between each deduplication tests.

The numbers

Small dataset: size: 17GB, Number of regular files: 38916, Average file size: 457KB

Without partial lookup

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes	9.21	15.14	0:08.59	111436	16MB
First run with deduplication	duperemove -rh data/ --hashfile=/tmp/hashes -d	10.32	19.88	0:14.82	113772	16MB
Second run without changes	duperemove -rh data/ --hashfile=/tmp/hashes	0.13	0.42	0:00.91	12544	16MB

With partial lookup

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial	67.10	20.47	1:07.35	118968	30MB
First run with deduplication	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial -d	65.96	23.38	1:12.99	123436	30MB
Second run without changes	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial	0.11	0.44	0:00.94	12912	30MB

Larger dataset (vm images): size: 48GB, Number of regular files: 6, Average file size: 8.1GB

Without partial lookup

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes	13.60	21.88	0:16.09	76656	2.6MB
First run with deduplication	duperemove -rh data/ --hashfile=/tmp/hashes -d	14.19	27.58	0:21.04	73952	548KB
Second run without changes	duperemove -rh data/ --hashfile=/tmp/hashes	0.00	0.01	0:00.02	5376	548KB

With partial lookup

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial	21.69	23.82	0:18.25	127356	35MB
First run with deduplication	duperemove -rh data/ --hashfile=/tmp/hashes -d --dedupe-option=partial	22.51	34.06	0:32.38	132116	35MB
Second run without changes	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial	0.00	0.01	0:00.02	5120	35MB

Largest dataset: size: 1.1TB, Number of regular files: 387930, Average file size: 3.3MB

Without partial lookup

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes	612.84	620.25	12:39.35	263440	166MB
Second run without changes	duperemove -rh data/ --hashfile=/tmp/hashes	250.40	12.10	4:28.45	129488	166MB

With partial lookup

The run was shutdown, all the files were scanned but the potentials duplicates were still printing.

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial	4776.97	1755.69	1:35:58	561272	968MB

Large number of identical small files dataset: size: 3.9G, Number of regular files: 1000000, Average file size: 1KB

Without partial lookup

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes	1654.13	209.21	30:01.68	212196	281M
First run with deduplication	duperemove -rh data/ --hashfile=/tmp/hashes -d	1157.57	764.88	22:49.95	247116	280M
Second run without changes	duperemove -rh data/ --hashfile=/tmp/hashes	2.68	9.43	0:14.03	131200	280M

With partial lookup

	Command	User time (seconds)	System time (seconds)	Elapsed (wall clock) time (h:mm:ss or m:ss)	Maximum resident set size (kbytes)	Hashfile size
First run without deduplication	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial
First run with deduplication	duperemove -rh data/ --hashfile=/tmp/hashes -d --dedupe-option=partial
Second run without changes	duperemove -rh data/ --hashfile=/tmp/hashes --dedupe-option=partial

Hardware details

Kernel/Distribution

3.84 [jack:~] lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux trixie/sid
Release:	n/a
Codename:	trixie
3.93 [jack:~] uname -a
Linux debian 6.5.0-4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.10-1 (2023-11-03) x86_64 GNU/Linux

CPU (Ryzen 3600, 12 x 3.60GHz)

3.93 [jack:~] cat /proc/cpuinfo 
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 23
model		: 113
model name	: AMD Ryzen 5 3600 6-Core Processor
stepping	: 0
microcode	: 0x8701021
cpu MHz		: 2794.327
cache size	: 512 KB
physical id	: 0
siblings	: 12
core id		: 0
cpu cores	: 6
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 16
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev sev_es
bugs		: sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass retbleed smt_rsb srso
bogomips	: 7186.05
TLB size	: 3072 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

    <12 cpus in total, redudant info removed>

Memory (32GB RAM 2666MHz C16)

  3.96 [jack:~] cat /proc/meminfo | head -10
  MemTotal:       32741508 kB
  MemFree:          952968 kB
  MemAvailable:   28113416 kB
  Buffers:           18360 kB
  Cached:         26950188 kB
  SwapCached:            0 kB
  Active:          4939996 kB
  Inactive:       25399956 kB
  Active(anon):    2798580 kB
  Inactive(anon):   772904 kB

Disk

(Micron 9300 MTFDHAL3T8TDP), ~1TB data on an xfs partition