zstd.Decode refuses to decode valid input.

[klauspost]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.12.1 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\mfiles\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=e:\gopath\
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=e:\temp\wintemp\go-build152196250=/tmp/go-build -gno-record-gcc-switches

Package version: 809b919c325d7887bff7bd876162af73db53e878 (v.1.4.0 tag)

What did you do?

Using standard zstd.Decompress function.

Reproducer: badref.zip - includes main function and sample data)

Using the commandline tool is fine:

zstd.exe -d 2aa9ddad029fa86d4e4831b22051a3b8414f015f.zst
2aa9ddad029fa86d4e4831b22051a3b8414f015f.zst: 137 bytes

What did you expect to see?

The length of the decompressed data (137 bytes)

What did you see instead?

panic: Corrupted block detected

[Viq111]

Hi @klauspost !

Thanks a lot for taking the time to run this through a fuzzer!
For this one, the zstd cli and the go library are not at feature parity (i.e: a payload encoded with one and decoded with the other will not usually) works. This is because the zstd cli adds some headers and implements more features (for example: checksums #43)
To confirm the behavior, I just tried your example with another bindings from a different author, the main one for python: https://pypi.org/project/zstd/

And I get the same results:

>>> import zstd
>>> with open("2aa9ddad029fa86d4e4831b22051a3b8414f015f.zst", "rb") as f:
...   a = f.read()
>>> zstd.decompress(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zstd.Error: Decompression error: Corrupted block detected

The other 2 examples that the fuzzer uncovered looks like bugs though (#60 for sure, #59 needs more investigation)