DATA RACE when going over results of sublist match and sub removed from sublist

[kozlovic]

This was seen when working on PR #638, and showed on an unrelated test. The new test added in the PR (TestRoutedQueueUnsubscribe) is reporting race at almost every run when running locally.

IvanMBP:gnatsd ivan$ go test -race -v -run=TestRoutedQueueUnsubscribe -count 10 ./server/
=== RUN   TestRoutedQueueUnsubscribe
==================
WARNING: DATA RACE
Write at 0x00c420496af8 by goroutine 31:
  github.com/nats-io/gnatsd/server.removeSubFromList()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/sublist.go:440 +0x100
  github.com/nats-io/gnatsd/server.(*Sublist).removeFromNode()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/sublist.go:460 +0x174
  github.com/nats-io/gnatsd/server.(*Sublist).Remove()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/sublist.go:378 +0x80a
  github.com/nats-io/gnatsd/server.(*client).unsubscribe()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/client.go:847 +0x27d
  github.com/nats-io/gnatsd/server.(*client).deliverMsg()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/client.go:1002 +0x708
  github.com/nats-io/gnatsd/server.(*client).processMsg()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/client.go:1142 +0x20b0
  github.com/nats-io/gnatsd/server.(*client).parse()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/parser.go:226 +0x1fd1
  github.com/nats-io/gnatsd/server.(*client).readLoop()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/client.go:284 +0xe93
  github.com/nats-io/gnatsd/server.(*Server).createRoute.func2()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/route.go:407 +0x41

Previous read at 0x00c420496af8 by goroutine 43:
  github.com/nats-io/gnatsd/server.(*client).processMsg()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/client.go:1198 +0x1296
  github.com/nats-io/gnatsd/server.(*client).parse()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/parser.go:226 +0x1fd1
  github.com/nats-io/gnatsd/server.(*client).readLoop()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/client.go:284 +0xe93
  github.com/nats-io/gnatsd/server.(*Server).createClient.func2()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/server.go:805 +0x41

Goroutine 31 (running) created at:
  github.com/nats-io/gnatsd/server.(*Server).startGoRoutine()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/server.go:1059 +0xba
  github.com/nats-io/gnatsd/server.(*Server).createRoute()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/route.go:407 +0x62a
  github.com/nats-io/gnatsd/server.(*Server).connectToRoute()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/route.go:784 +0x4cc
  github.com/nats-io/gnatsd/server.(*Server).solicitRoutes.func1()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/route.go:798 +0x4f

Goroutine 43 (running) created at:
  github.com/nats-io/gnatsd/server.(*Server).startGoRoutine()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/server.go:1059 +0xba
  github.com/nats-io/gnatsd/server.(*Server).createClient()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/server.go:805 +0x673
  github.com/nats-io/gnatsd/server.(*Server).AcceptLoop.func2()
      /Users/ivan/dev/go/src/github.com/nats-io/gnatsd/server/server.go:475 +0x58
==================
If this test is brought to master, the same issue happens, so this is not specific to that branch.