ipfs crashes on Add

[alikic]

Version information:

go-ipfs version: 0.4.4-dev-e10453b
Repo version: 4
System version: amd64/linux
Golang version: go1.7

Type:

Bug

Priority:

P0

Description:

ipfs crashes on Add. We found this after taking the latest dev version, under very light use. We are able to replicate by stress test.

Steps to reproduce:
rm -rf $GOPATH/bin $GOPATH/pkg $GOPATH/src
go get -d github.com/ipfs/go-ipfs
cd $GOPATH/src/github.com/ipfs/go-ipfs
make install
go get github.com/securekey/ipfs-performance-test/ipfs-pt-agent
go get github.com/securekey/ipfs-performance-test/ipfs-pt-manager
cd $GOPATH/src/github.com/securekey/ipfs-performance-test/ipfs-pt-manager/test
mkdir ipfs
export IPFS_PATH=$GOPATH/src/github.com/securekey/ipfs-performance-test/ipfs-pt-manager/test/ipfs
ipfs init
ipfs daemon &
ipfs-pt-agent -c a1.json

In another shell, run the manager:

ipfs-pt-manager -c mlocal.json -o out

After 0-4%, ipfs crashes with the following:

panic: runtime error: index out of range

goroutine 94 [running]:
panic(0xbd8760, 0xc4200100e0)
/usr/local/go17/src/runtime/panic.go:500 +0x1a1
github.com/ipfs/go-ipfs/merkledag.(_Node).getPBNode(0xc4230218b0, 0xc420194c88)
/root/go17/src/github.com/ipfs/go-ipfs/merkledag/coding.go:61 +0x33f
github.com/ipfs/go-ipfs/merkledag.(_Node).Marshal(0xc4230218b0, 0xc42344bb60, 0xc42344bb60, 0x10f4680, 0xc42344bb60, 0x2)
/root/go17/src/github.com/ipfs/go-ipfs/merkledag/coding.go:45 +0x2f
github.com/ipfs/go-ipfs/merkledag.(_Node).EncodeProtobuf(0xc4230218b0, 0x0, 0x3, 0x22, 0x22, 0x22, 0x30)
/root/go17/src/github.com/ipfs/go-ipfs/merkledag/coding.go:80 +0xb3
github.com/ipfs/go-ipfs/merkledag.(_Node).Multihash(0xc4230218b0, 0xc420194d98, 0x449985, 0xc420194da8)
/root/go17/src/github.com/ipfs/go-ipfs/merkledag/node.go:271 +0x34
github.com/ipfs/go-ipfs/merkledag.(_Node).Key(0xc4230218b0, 0xc420267f80, 0x0)
/root/go17/src/github.com/ipfs/go-ipfs/merkledag/node.go:249 +0x2b
github.com/ipfs/go-ipfs/exchange/bitswap.(_Bitswap).provideCollector(0xc4201f0dc0, 0x10f6d00, 0xc4200622c0)
/root/go17/src/github.com/ipfs/go-ipfs/exchange/bitswap/workers.go:138 +0x2c4
github.com/ipfs/go-ipfs/exchange/bitswap.(_Bitswap).startWorkers.func4(0x10fb7e0, 0xc42033a7e0)
/root/go17/src/github.com/ipfs/go-ipfs/exchange/bitswap/workers.go:40 +0x3c
gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess.(_process).Go.func1(0xc4200d5240, 0xc42033a7e0, 0xc42033a900)
/root/go17/src/gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess/impl-mutex.go:112 +0x3c
created by gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess.(*process).Go
/root/go17/src/gx/ipfs/QmSF8fPo3jgVBAy8fpdjjYqgG87dkJgUprRBHRd2tmfgpP/goprocess/impl-mutex.go:115 +0x375
panic: send on closed channel

goroutine 29 [running]:
panic(0x7b1140, 0xc420013b00)
/usr/local/go17/src/runtime/panic.go:500 +0x1a1
github.com/securekey/ipfs-performance-test/ipfs-pt-agent/agent.(_Agent).Add.func2(0xc42017c8f0, 0xc420126af0)
/root/go17/src/github.com/securekey/ipfs-performance-test/ipfs-pt-agent/agent/add.go:52 +0x5c
created by github.com/securekey/ipfs-performance-test/ipfs-pt-agent/agent.(_Agent).Add
/root/go17/src/github.com/securekey/ipfs-performance-test/ipfs-pt-agent/agent/add.go:54 +0x23c
[1]+ Exit 2 ipfs daemon

[alikic]

You can speed up the test by setting --pin=false. go-ipfs-api add() doesn't support this option yet. so you have to modifie shell.go in $GOPATH/src to hardcode this option just for this test:

vi $GOPATH/src/github.com/ipfs/go-ipfs-api/shell.go

    req := NewRequest(s.url, "add")
    req.Body = fileReader
    req.Opts["progress"] = "false"
    req.Opts["pin"] = "false" <--- add this line

and then

go install github.com/ipfs/go-ipfs-api
go install github.com/securekey/ipfs-performance-test/ipfs-pt-agent

Now just start the daemon, the agent and the manager.

[chriscool]

Thanks! This may be what we also see on issue #3199.

[alikic]

This instability was introduced with 3babf40.

Earlier versions are stable. Later versions all have the same problem, some of them failing sooner and some later.

[Kubuxu]

@alikic thank you for tracking it.

[whyrusleeping]

@alikic Yeah, thanks for tracking this down! I'll get this resolved ASAP

[alikic]

@whyrusleeping Great, thanks! You should be able to verify the fix by running the performance test as described above. If you run the test with pinning enabled (which is hard-coded in current go-ipfs-api), you will encounter the "freezing" issue described at #3131. In my latest test (using the same parameters as here), it happens after about 8000 adds.

[whyrusleeping]

Cool, this has been resolved. thanks @alikic