Investigate using a zstd library consistent with Lotus #850
Assignees
Milestone
Comments
masih
added a commit
that referenced
this issue
Jan 27, 2025
Use `DataDog/zstd` library for: * a more performant implementation * consistent dependency with the one used in Lotus See Benchamark results below. Before: ``` BenchmarkCborEncoding BenchmarkCborEncoding-12 467352 2376 ns/op 14680 B/op 10 allocs/op BenchmarkCborDecoding BenchmarkCborDecoding-12 104410 11347 ns/op 14944 B/op 27 allocs/op BenchmarkZstdEncoding BenchmarkZstdEncoding-12 286735 3897 ns/op 46748 B/op 12 allocs/op BenchmarkZstdDecoding BenchmarkZstdDecoding-12 110794 10783 ns/op 28512 B/op 28 allocs/op ``` After: ``` BenchmarkCborEncoding BenchmarkCborEncoding-12 439345 2436 ns/op 14680 B/op 10 allocs/op BenchmarkCborDecoding BenchmarkCborDecoding-12 105511 11395 ns/op 14944 B/op 27 allocs/op BenchmarkZstdEncoding BenchmarkZstdEncoding-12 247182 4790 ns/op 28248 B/op 11 allocs/op BenchmarkZstdDecoding BenchmarkZstdDecoding-12 113475 10694 ns/op 20617 B/op 28 allocs/op ``` Fixes: #850 #849
masih
added a commit
that referenced
this issue
Jan 27, 2025
Use `DataDog/zstd` library for: * a more performant implementation * consistent dependency with the one used in Lotus See Benchamark results below. Before: ``` BenchmarkCborEncoding BenchmarkCborEncoding-12 467352 2376 ns/op 14680 B/op 10 allocs/op BenchmarkCborDecoding BenchmarkCborDecoding-12 104410 11347 ns/op 14944 B/op 27 allocs/op BenchmarkZstdEncoding BenchmarkZstdEncoding-12 286735 3897 ns/op 46748 B/op 12 allocs/op BenchmarkZstdDecoding BenchmarkZstdDecoding-12 110794 10783 ns/op 28512 B/op 28 allocs/op ``` After: ``` BenchmarkCborEncoding BenchmarkCborEncoding-12 439345 2436 ns/op 14680 B/op 10 allocs/op BenchmarkCborDecoding BenchmarkCborDecoding-12 105511 11395 ns/op 14944 B/op 27 allocs/op BenchmarkZstdEncoding BenchmarkZstdEncoding-12 247182 4790 ns/op 28248 B/op 11 allocs/op BenchmarkZstdDecoding BenchmarkZstdDecoding-12 113475 10694 ns/op 20617 B/op 27 allocs/op ``` Fixes: #850 #849
masih
added a commit
that referenced
this issue
Jan 28, 2025
Pool the decode buffers and set strict max cap allocation for zstd decompressor. See Benchamark results below. Before: ``` BenchmarkCborEncoding BenchmarkCborEncoding-12 467352 2376 ns/op 14680 B/op 10 allocs/op BenchmarkCborDecoding BenchmarkCborDecoding-12 104410 11347 ns/op 14944 B/op 27 allocs/op BenchmarkZstdEncoding BenchmarkZstdEncoding-12 286735 3897 ns/op 46748 B/op 12 allocs/op BenchmarkZstdDecoding BenchmarkZstdDecoding-12 110794 10783 ns/op 28512 B/op 28 allocs/op ``` After: ``` BenchmarkCborEncoding BenchmarkCborEncoding-12 436754 2383 ns/op 14680 B/op 10 allocs/op BenchmarkCborDecoding BenchmarkCborDecoding-12 106809 11280 ns/op 14944 B/op 27 allocs/op BenchmarkZstdEncoding BenchmarkZstdEncoding-12 294043 3918 ns/op 46746 B/op 12 allocs/op BenchmarkZstdDecoding BenchmarkZstdDecoding-12 114854 10747 ns/op 18314 B/op 27 allocs/op ``` Fixes: #850 #849
|
Cross posting lotus benchmark results for visibility: filecoin-project/lotus#12852 (comment) |
|
Lotus PR to switch to pure go decoder, same as the one currently used by F3: filecoin-project/lotus#12857 Closing this as there are no more actionables here. |
|
Thanks for driving this to close with data @masih. Nice work! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Lotus uses Datadog/zstd (which internally wraps C libraries) but go-f3 uses a pure-go implementation of zstd.
The C version is faster, specifically for non-streaming version. Ideally, we want to use a consistent dependency across both Lotus and go-f3 anyway.
Thanks to @ribasushi for pointing this out 🙏
The text was updated successfully, but these errors were encountered: