runtime: segfault during test using -coverpkg and -race

[yonderblue]

What version of Go are you using (go version)?

go1.11.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Linux amd64

What did you do?

go test -c -o aTestBinary -coverpkg package1,package2,package3 -covermode atomic -race -ldflags '-w -s -extldflags "-static -Wl,--no-export-dynamic"'
./aTestBinary -test.v -test.run . -test.coverprofile coverout

What did you expect to see?

Success

What did you see instead?

unexpected fault address 0x19600
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x19600 pc=0x12a0389]

goroutine 63887 [running]:
runtime: unexpected return pc for runtime.throw called from 0x5
stack: frame={sp:0x7f0386ffca70, fp:0x7f0386ffcaa0} stack=[0xc0021f4000,0xc0021f6000)

runtime.throw(0x5, 0x7f0386ffcae0)
	/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0x7f0386ffcaa0 sp=0x7f0386ffca70 pc=0x497d42
created by ...

goroutine 1 [chan receive]:
testing.tRunner.func1(0xc000112100)
	/usr/local/go/src/testing/testing.go:803 +0x2e4
testing.tRunner(0xc000112100, 0xc00011bc98)
	/usr/local/go/src/testing/testing.go:831 +0x17f
testing.runTests(0xc0000c4060, 0x410d5c0, 0xa5, 0xa5, 0xc0000bc000)
	/usr/local/go/src/testing/testing.go:1117 +0x4ef
testing.(*M).Run(0xc000232200, 0x0)
	/usr/local/go/src/testing/testing.go:1034 +0x2ef
.../....TestMain(0xc000232200)
	/go/src/....go:36 +0x123
main.main()
	_testmain.go:962 +0x33f

goroutine 50 [chan receive]:
testing.(*testContext).waitParallel(0xc0000ad1d0)
	/usr/local/go/src/testing/testing.go:926 +0x15c
testing.(*T).Parallel(0xc000112200)
	/usr/local/go/src/testing/testing.go:733 +0x3e5
...

Probably related #23882?

[agnivade]

Do you have a shortened reproducer or perhaps a sample repo for us to test this out and reproduce it ourselves ?

[yonderblue]

I don't have a repro as it happens in the middle of a larger suite with a number of parallel tests running. However it appears to be related mainly to the race detector. I have not seen the faults without the race detector and without the coverpkg option. For example here is another one, with race detector but without any coverage:

unexpected fault address 0x2600
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x2600 pc=0x12629d9]

goroutine 424974 [running]:
runtime: unexpected return pc for runtime.throw called from 0x6
stack: frame={sp:0x7f0bf7ffea70, fp:0x7f0bf7ffeaa0} stack=[0xc002452000,0xc00245a000)

runtime.throw(0x6, 0x7f0bf7ffeae0)
	/usr/local/go/src/runtime/panic.go:608 +0x72 fp=0x7f0bf7ffeaa0 sp=0x7f0bf7ffea70 pc=0x497d42
created by testing.(*B).RunParallel
	/usr/local/go/src/testing/benchmark.go:619 +0x256

[yonderblue]

@agnivade Anyway I can narrow down the PC address to help point in a right direction?

[agnivade]

/cc @ianlancetaylor for further comments.

[ianlancetaylor]

CC @aclements @dvyukov

[ianlancetaylor]

The signal report is giving you a PC address. Can you tell us what is at that address?

[dballard]

go 1.10.3
I am incorporating https://github.com/cretz/bine into https://git.openprivacy.ca/cwtch.im/cwtch/

~/s/g/s/c/c/peer$ go test
PASS
ok  	cwtch.im/cwtch/peer	0.003s

~/s/g/s/c/c/peer$ go test -coverprofile=peer.cover.out
PASS
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x647958]

goroutine 23 [running]:
github.com/cretz/bine/tor.(*Tor).EnableNetwork(0x0, 0x73e520, 0xc4200a4050, 0x1, 0x0, 0x0)
	/home/dan/src/go/src/github.com/cretz/bine/tor/tor.go:367 +0x78
github.com/cretz/bine/tor.(*Tor).Dialer(0x0, 0x0, 0x0, 0xc420132e38, 0x0, 0x0, 0x0)
	/home/dan/src/go/src/github.com/cretz/bine/tor/dialer.go:55 +0x3c3
git.openprivacy.ca/openprivacy/libricochet-go/utils.(*NetworkResolver).Resolve(0xc420041ed8, 0x0, 0xc4201025c0, 0x38, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/dan/src/go/src/git.openprivacy.ca/openprivacy/libricochet-go/utils/networkresolver.go:44 +0x22c
git.openprivacy.ca/openprivacy/libricochet-go.Open(0x0, 0xc4201025c0, 0x38, 0x0, 0x0, 0x0)
	/home/dan/src/go/src/git.openprivacy.ca/openprivacy/libricochet-go/ricochet.go:18 +0x56
cwtch.im/cwtch/peer/connections.(*PeerPeerConnection).Run(0xc4200a87d0, 0x38, 0x8c7760)
	/home/dan/src/go/src/cwtch.im/cwtch/peer/connections/peerpeerconnection.go:113 +0x5e
created by cwtch.im/cwtch/peer/connections.(*Manager).ManagePeerConnection
	/home/dan/src/go/src/cwtch.im/cwtch/peer/connections/connectionsmanager.go:39 +0x17b
exit status 2
FAIL	cwtch.im/cwtch/peer	0.009s

It seems to be a problem that my Init function which spawns a go thread takes a *tor.Tor argument, which for these tests is never used (just testing paths where we don't make connections, so it's never invoked) so I pass in a nil. The test runs fine, it seems the cover is attempting to follow the uncalled code

test

err = alice.TrustPeer(bobOnion)
if err == nil {
	t.Errorf("trust peer should fail since alice does not know about bob")
}

func (cp *cwtchPeer) TrustPeer(peer string) error {
	err := cp.Profile.TrustPeer(peer)
	if err == nil {
		cp.PeerWithOnion(peer)
	}
	return err
}

PeerWithOnion should never be called, and the test passes, but as you can see from the segfault trace, it has tracked into PeerWithOnion which calls ManagePeerConections which starts a goroutine go ppc.Run() which is where the above stack trace starts. Then it decends into our dependancy and segfaults.

What is really weird is since we've supplied nil it... shoudn't go that far? but the cover check decends into the tor.Tor object ok, but then it seems to be at

	if vals, err := t.Control.GetConf("DisableNetwork"); err != nil {

when the tor.Tor (our nil) object tries to access a field it finally explodes and fails

hope this is helpful.