[3.4] WARNING: DATA RACE between mvcc.(*store).setupMetricsReporter and mvcc.(*store).restore

[ahrtr]

I see the following error in 3.4 pipeline,

==================
WARNING: DATA RACE
Read at 0x00c003404a00 by goroutine 943:
  go.etcd.io/etcd/mvcc.(*store).setupMetricsReporter.func5()
      /home/runner/work/etcd/etcd/mvcc/kvstore.go:600 +0xa9
  go.etcd.io/etcd/mvcc.glob..func9()
      /home/runner/work/etcd/etcd/mvcc/metrics.go:277 +0x95
  github.com/prometheus/client_golang/prometheus.(*valueFunc).Write()
      /home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/value.go:72 +0x52
  github.com/prometheus/client_golang/prometheus.processMetric()
      /home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:594 +0x131
  github.com/prometheus/client_golang/prometheus.(*Registry).Gather()
--
  net/http.HandlerFunc.ServeHTTP()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:2050 +0x51
  github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1()
      /home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:40 +0xf6
  net/http.HandlerFunc.ServeHTTP()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:2050 +0x51
  github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1()
      /home/runner/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:100 +0xfd
  net/http.HandlerFunc.ServeHTTP()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:2050 +0x51
  net/http.(*ServeMux).ServeHTTP()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:2429 +0xaf
  go.etcd.io/etcd/etcdserver/api/v2http.requestLogger.func1()
      /home/runner/work/etcd/etcd/etcdserver/api/v2http/http.go:91 +0x461
  net/http.HandlerFunc.ServeHTTP()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:2050 +0x51
  net/http.serverHandler.ServeHTTP()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:2868 +0xca
  net/http.(*conn).serve()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:1933 +0x87d
Previous write at 0x00c003404a00 by goroutine 1135:
  go.etcd.io/etcd/mvcc.(*store).restore()
      /home/runner/work/etcd/etcd/mvcc/kvstore.go:424 +0x764
  go.etcd.io/etcd/mvcc.(*store).Restore()
      /home/runner/work/etcd/etcd/mvcc/kvstore.go:366 +0x594
  go.etcd.io/etcd/mvcc.(*watchableStore).Restore()
      /home/runner/work/etcd/etcd/mvcc/watchable_store.go:198 +0xe6
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applySnapshot()
      /home/runner/work/etcd/etcd/etcdserver/server.go:1240 +0xf31
  go.etcd.io/etcd/etcdserver.(*EtcdServer).applyAll()
      /home/runner/work/etcd/etcd/etcdserver/server.go:1135 +0x6f
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run.func8()
      /home/runner/work/etcd/etcd/etcdserver/server.go:1072 +0x53
  go.etcd.io/etcd/pkg/schedule.(*fifo).run()
      /home/runner/work/etcd/etcd/pkg/schedule/schedule.go:157 +0x12a
Goroutine 943 (running) created at:
  net/http.(*Server).Serve()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/server.go:2994 +0x644
  net/http/httptest.(*Server).goServe.func1()
      /opt/hostedtoolcache/go/1.16.15/x64/src/net/http/httptest/server.go:308 +0xd8
Goroutine 1135 (running) created at:
  go.etcd.io/etcd/pkg/schedule.NewFIFOScheduler()
      /home/runner/work/etcd/etcd/pkg/schedule/schedule.go:70 +0x2d8
  go.etcd.io/etcd/etcdserver.(*EtcdServer).run()
      /home/runner/work/etcd/etcd/etcdserver/server.go:957 +0x344
==================

Refer to https://github.com/etcd-io/etcd/runs/7491976783?check_suite_focus=true

[ahrtr]

@SimFG Are you interested and have bandwidth to have a deep dive into this issue?

[SimFG]

@SimFG Are you interested and have bandwidth to have a deep dive into this issue?

Ok, I can try to do it.

[ahrtr]

Resolved by #14279