move go-libp2p-autonat to p2p/host/autonat

[marten-seemann]

There's no reason autonat should live in a separate repo.

This is also a prerequisite for moving the swarm into this repo, as autonat depends on go-libp2p-swarm/testing.

[vyzo]

do it!

[marten-seemann]

Looks like tests are flaky. Let's fix in the autonat repo and then move it here.

[marten-seemann]

The problem is not that the test are flaky on their own - they're only flaky because go test runs tests in packages in parallel. I could imagine that the tricks we play for the holepunching test mess with the public / private detection here.

[vyzo]

sounds plausible; any wsy to put constraints on what runs in parallel?

[mvdan]

I do think we can blame at least one test for being flaky. At least it's flaky when it runs in parallel to itself, even under separate binaries and processes. Enter the new ./p2p/host/autonat package, or the old root package, with Go 1.17.5, and run:

go test -c -o pkg.test && stress ./pkg.test -test.run=TestAutoNATPrivate

In both the old and new repos, I get a failure rate of just under 5%. Low enough that the flake probably didn't cause issues in the smaller module, perhaps.

$ go test -c -o pkg.test && stress ./pkg.test -test.run=TestAutoNATPrivate
go: downloading github.com/libp2p/go-libp2p-core v0.12.0
5s: 32 runs so far, 0 failures

/tmp/go-stress-20211220T193854-3125905374
--- FAIL: TestAutoNATPrivate (2.01s)
    autonat_test.go:118: unexpected NAT status: 0
FAIL


ERROR: exit status 1


/tmp/go-stress-20211220T193854-1717313164
--- FAIL: TestAutoNATPrivate (2.01s)
    autonat_test.go:118: unexpected NAT status: 0
FAIL


ERROR: exit status 1

10s: 64 runs so far, 2 failures (3.12%)

I'm not sure if the test is simply flakey by design, or if it starts to flake when run in parallel with other tests in other processes - perhaps it's using the network in some way that clashes globally, such as listening on ports or sending broadcast messages.

In any case, I don't think we can blame go test ./... for this one - it's just that, the more tests you run in parallel, the higher the chances that flakes will show up :)

[mvdan]

I should have mentioned that the tool in question is https://pkg.go.dev/golang.org/x/tools/cmd/stress. It's just a small tool to run many parallel invocations of a command, and find failures. One of its main intended use cases is to uncover flakes.

If you struggle to reproduce, note that you can give extra flags to stress to increase the parallelism and much more.

[marten-seemann]

@mvdan Thank you for helping debug this.
Turns out that this was an actual race condition (only in the test code though), which is highly timing-dependent: libp2p/go-libp2p-autonat#120.
I suppose the higher CPU load caused by running all packages in parallel caused this to be triggered more reliably here than in go-libp2p-autonat.

I will update this PR once this fix has been merged.