all: go tool dist test -msan fails many tests

[qiulaidongfeng]

What version of Go are you using (go version)?

$ go version
go version devel go1.22-631a6c2 Fri Nov 17 23:34:11 2023 +0000 linux/amd64

Does this issue reproduce with the latest release?

yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://goproxy.cn,direct'
GOROOT='/root/.go/gotip'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/root/.go/gotip/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.22-631a6c2 Fri Nov 17 23:34:11 2023 +0000'
GCCGO='gccgo'
GOAMD64='v4'
AR='ar'
CC='clang'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build853797895=/tmp/go-build -gno-record-gcc-switches'

What did you do?

go tool dist test -msan fail

What did you expect to see?

test pass.

What did you see instead?

##### Test execution environment.
# GOARCH: amd64
# CPU: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics
# GOOS: linux
# OS Version: Linux 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64

##### Testing packages.
ok      archive/tar     0.510s
ok      archive/zip     0.628s
ok      bufio   0.343s
ok      bytes   0.650s
ok      cmp     0.006s
ok      compress/bzip2  0.682s
ok      compress/flate  4.046s
ok      compress/gzip   12.570s
ok      compress/lzw    0.099s
ok      compress/zlib   0.099s
ok      container/heap  0.018s
ok      container/list  0.018s
ok      container/ring  0.021s
ok      context 0.056s
ok      crypto  0.021s
ok      crypto/aes      0.035s
ok      crypto/cipher   0.043s
ok      crypto/des      0.035s
ok      crypto/dsa      0.017s
ok      crypto/ecdh     0.459s
ok      crypto/ecdsa    0.277s
ok      crypto/ed25519  0.716s
ok      crypto/elliptic 0.078s
ok      crypto/hmac     0.023s
ok      crypto/internal/alias   0.024s
ok      crypto/internal/bigmod  6.189s
ok      crypto/internal/boring  0.023s
ok      crypto/internal/boring/bcache   0.308s
ok      crypto/internal/edwards25519    1.277s
ok      crypto/internal/edwards25519/field      0.070s
ok      crypto/internal/nistec  1.935s
ok      crypto/internal/nistec/fiat     0.034s [no tests to run]
ok      crypto/md5      0.076s
ok      crypto/rand     0.387s
ok      crypto/rc4      0.755s
ok      crypto/rsa      2.609s
ok      crypto/sha1     0.043s
ok      crypto/sha256   0.024s
ok      crypto/sha512   0.032s
ok      crypto/subtle   0.500s
ok      crypto/tls      5.481s
ok      crypto/x509     2.541s
ok      database/sql    0.526s
ok      database/sql/driver     0.013s
ok      debug/buildinfo 0.017s
ok      debug/dwarf     0.060s
ok      debug/elf       1.113s
ok      debug/gosym     0.398s
ok      debug/macho     0.039s
ok      debug/pe        0.041s
ok      debug/plan9obj  0.010s
ok      embed   0.010s [no tests to run]
ok      embed/internal/embedtest        0.009s
ok      encoding/ascii85        0.025s
ok      encoding/asn1   0.037s
ok      encoding/base32 0.045s
ok      encoding/base64 0.027s
ok      encoding/binary 0.017s
ok      encoding/csv    0.060s
ok      encoding/gob    20.500s
ok      encoding/hex    0.025s
ok      encoding/json   1.030s
ok      encoding/pem    1.298s
ok      encoding/xml    0.202s
ok      errors  0.032s
ok      expvar  0.037s
ok      flag    0.206s
ok      fmt     0.746s
ok      go/ast  0.045s
ok      go/build        5.842s
ok      go/build/constraint     0.038s
ok      go/constant     0.016s
ok      go/doc  0.177s
ok      go/doc/comment  2.288s
ok      go/format       0.058s
ok      go/importer     0.494s
ok      go/internal/gccgoimporter       0.042s
ok      go/internal/gcimporter  9.275s
ok      go/internal/srcimporter 18.086s
ok      go/parser       1.301s
ok      go/printer      0.914s
ok      go/scanner      0.020s
ok      go/token        0.066s
ok      go/types        18.392s
ok      go/version      0.022s
ok      hash    0.013s
ok      hash/adler32    0.037s
ok      hash/crc32      0.058s
ok      hash/crc64      0.021s
ok      hash/fnv        0.012s
ok      hash/maphash    0.555s
ok      html    0.034s
ok      html/template   0.201s
ok      image   0.603s
ok      image/color     0.042s
ok      image/draw      0.798s
ok      image/gif       1.253s
ok      image/jpeg      1.996s
ok      image/png       3.973s
ok      index/suffixarray       2.182s
ok      internal/abi    0.145s
ok      internal/buildcfg       0.006s
ok      internal/coverage/cformat       0.034s
ok      internal/coverage/cmerge        0.025s
ok      internal/coverage/pods  0.024s
ok      internal/coverage/slicereader   0.011s
ok      internal/coverage/slicewriter   0.033s
ok      internal/coverage/test  0.049s
ok      internal/cpu    0.028s
ok      internal/dag    0.020s
ok      internal/diff   0.050s
ok      internal/fmtsort        0.028s
ok      internal/fuzz   0.037s
ok      internal/godebug        1.711s
ok      internal/godebugs       0.014s
ok      internal/gover  0.020s
ok      internal/intern 1.176s
ok      internal/itoa   0.047s
ok      internal/platform       0.505s
ok      internal/poll   0.498s
ok      internal/profile        0.008s
ok      internal/reflectlite    0.011s
ok      internal/safefilepath   0.008s
ok      internal/saferio        0.095s
ok      internal/singleflight   0.030s
ok      internal/testenv        0.247s
ok      internal/trace  0.546s
ok      internal/trace/v2       11.447s
ok      internal/types/errors   0.785s
ok      internal/unsafeheader   0.036s
ok      internal/xcoff  0.075s
ok      internal/zstd   0.059s
ok      io      0.035s
ok      io/fs   0.199s
ok      io/ioutil       0.022s
ok      log     0.035s
--- FAIL: TestAlloc (0.01s)
    --- FAIL: TestAlloc/attrs6 (0.00s)
        logger_test.go:312: got 2 allocs, want 1
    --- FAIL: TestAlloc/attrs9 (0.00s)
        logger_test.go:319: got 2 allocs, want 1
FAIL
FAIL    log/slog        0.135s
ok      log/slog/internal/benchmarks    0.034s
ok      log/slog/internal/buffer        0.041s
ok      log/syslog      1.243s
ok      maps    0.028s
ok      math    0.018s
ok      math/big        4.115s
ok      math/bits       0.035s
ok      math/cmplx      0.022s
ok      math/rand       1.334s
ok      math/rand/v2    1.775s
ok      mime    0.026s
ok      mime/multipart  4.014s
ok      mime/quotedprintable    0.091s
# runtime
# [runtime]
runtime/msan_amd64.s:31:1: [amd64] domsanread: unknown variable size; offset 8 is sz+8(FP)
runtime/msan_amd64.s:40:1: [amd64] msanwrite: unknown variable size; offset 8 is sz+8(FP)
runtime/msan_amd64.s:48:1: [amd64] msanmalloc: unknown variable size; offset 8 is sz+8(FP)
runtime/msan_amd64.s:56:1: [amd64] msanfree: unknown variable size; offset 8 is sz+8(FP)
runtime/msan_amd64.s:65:1: [amd64] msanmove: unknown variable size; offset 16 is sz+16(FP)
ok      net     9.057s
FAIL    runtime [build failed]
ok      net/http        30.704s
ok      net/http/cgi    0.571s
ok      net/http/cookiejar      0.012s
ok      net/http/fcgi   0.216s
ok      net/http/httptest       0.333s
ok      net/http/httptrace      0.009s
ok      net/http/httputil       0.681s
ok      net/http/internal       0.010s
ok      net/http/internal/ascii 0.009s
ok      net/http/pprof  5.160s
ok      net/internal/socktest   0.009s
ok      net/mail        0.020s
ok      net/netip       0.322s
ok      net/rpc 0.043s
ok      net/rpc/jsonrpc 0.012s
ok      net/smtp        0.090s
ok      net/textproto   0.052s
ok      net/url 0.014s
ok      os      1.841s
ok      os/exec 0.489s
ok      os/exec/internal/fdtest 0.030s
ok      os/signal       2.317s
ok      os/user 0.021s
ok      path    0.033s
ok      path/filepath   0.049s
ok      plugin  0.006s
ok      reflect 0.364s
ok      regexp  2.793s
ok      regexp/syntax   4.585s
ok      runtime/cgo     0.009s
ok      runtime/coverage        0.009s
ok      runtime/debug   0.067s
ok      runtime/internal/atomic 0.027s
ok      runtime/internal/math   0.009s
ok      runtime/internal/sys    0.008s
ok      runtime/internal/syscall        0.008s
ok      runtime/internal/wasitest       0.009s
ok      runtime/metrics 0.024s
ok      runtime/pprof   12.102s
ok      runtime/trace   5.991s
--- FAIL: TestGrow (0.00s)
    slices_test.go:915: Grow should allocate once when given insufficient capacity; allocated 2 times
--- FAIL: TestConcat (0.02s)
    slices_test.go:1263: Concat([[1]]) allocated 2 times; want 1
    slices_test.go:1263: Concat([[1] [2]]) allocated 2 times; want 1
    slices_test.go:1263: Concat([[1] [] [2]]) allocated 2 times; want 1
FAIL
FAIL    slices  0.369s
ok      sort    0.640s
ok      strconv 4.124s
ok      strings 0.225s
ok      sync    0.490s
ok      sync/atomic     0.721s
ok      syscall 1.983s
ok      testing 1.320s
ok      testing/fstest  0.011s
ok      testing/iotest  0.020s
ok      testing/quick   0.127s
ok      testing/slogtest        0.010s
ok      text/scanner    0.018s
ok      text/tabwriter  0.010s
ok      text/template   0.064s
ok      text/template/parse     0.015s
ok      time    1.950s
ok      unicode 0.011s
ok      unicode/utf16   0.011s
ok      unicode/utf8    0.022s
ok      cmd/addr2line   0.656s
ok      cmd/api 0.591s
ok      cmd/asm/internal/asm    1.834s
ok      cmd/asm/internal/lex    0.018s
ok      cmd/cgo/internal/swig   0.020s
ok      cmd/cgo/internal/test   1.731s
ok      cmd/cgo/internal/testcarchive   0.009s
ok      cmd/cgo/internal/testcshared    0.012s
ok      cmd/cgo/internal/testerrors     12.163s
ok      cmd/cgo/internal/testfortran    0.017s
ok      cmd/cgo/internal/testgodefs     2.065s
ok      cmd/cgo/internal/testlife       2.017s
ok      cmd/cgo/internal/testnocgo      0.013s
ok      cmd/cgo/internal/testplugin     0.011s
ok      cmd/cgo/internal/testsanitizers 21.174s
ok      cmd/cgo/internal/testshared     0.012s
ok      cmd/cgo/internal/testso 5.208s
ok      cmd/cgo/internal/teststdio      5.404s
ok      cmd/cgo/internal/testtls        0.011s
ok      cmd/compile/internal/abt        0.035s
ok      cmd/compile/internal/amd64      0.015s
ok      cmd/compile/internal/base       0.026s
ok      cmd/compile/internal/compare    0.017s
ok      cmd/compile/internal/devirtualize       0.023s
ok      cmd/compile/internal/dwarfgen   0.604s
ok      cmd/compile/internal/importer   1.409s
ok      cmd/compile/internal/inline/inlheur     0.579s
ok      cmd/compile/internal/ir 0.012s
ok      cmd/compile/internal/logopt     0.390s
ok      cmd/compile/internal/loopvar    39.697s
ok      cmd/compile/internal/noder      0.008s
ok      cmd/compile/internal/reflectdata        0.007s [no tests to run]
ok      cmd/compile/internal/ssa        38.775s
ok      cmd/compile/internal/syntax     0.293s
--- FAIL: TestAppendOfMake (0.00s)
    issue53888_test.go:22: got 1.000000 allocs, want 0
    issue53888_test.go:31: got 1.000000 allocs, want 0
    issue53888_test.go:37: got 1.000000 allocs, want 0
    issue53888_test.go:43: got 1.000000 allocs, want 0
FAIL
FAIL    cmd/compile/internal/test       9.050s
ok      cmd/compile/internal/typecheck  0.839s
ok      cmd/compile/internal/types      0.012s
ok      cmd/compile/internal/types2     16.011s
ok      cmd/covdata     0.013s
ok      cmd/cover       3.194s
ok      cmd/dist        0.011s
ok      cmd/distpack    0.012s
ok      cmd/doc 0.788s
ok      cmd/fix 3.418s
ok      cmd/go  37.669s
ok      cmd/go/internal/auth    0.012s
ok      cmd/go/internal/cache   0.274s
ok      cmd/go/internal/cfg     0.035s [no tests to run]
ok      cmd/go/internal/envcmd  0.148s
ok      cmd/go/internal/fsys    0.124s
ok      cmd/go/internal/generate        0.037s
ok      cmd/go/internal/gover   0.023s
ok      cmd/go/internal/imports 0.090s
ok      cmd/go/internal/load    0.075s
ok      cmd/go/internal/lockedfile      0.417s
ok      cmd/go/internal/lockedfile/internal/filelock    0.131s
ok      cmd/go/internal/modfetch        0.051s
ok      cmd/go/internal/modfetch/codehost       2.277s
ok      cmd/go/internal/modfetch/zip_sum_test   0.045s
ok      cmd/go/internal/modindex        1.259s
ok      cmd/go/internal/modload 0.057s
ok      cmd/go/internal/mvs     0.026s
ok      cmd/go/internal/par     0.065s
ok      cmd/go/internal/str     0.026s
ok      cmd/go/internal/test    0.094s
ok      cmd/go/internal/toolchain       0.059s
ok      cmd/go/internal/vcs     0.047s
ok      cmd/go/internal/vcweb   0.054s
ok      cmd/go/internal/vcweb/vcstest   2.259s
ok      cmd/go/internal/web     0.026s
ok      cmd/go/internal/work    0.288s
ok      cmd/gofmt       0.234s
ok      cmd/internal/archive    2.150s
ok      cmd/internal/bootstrap_test     0.061s
ok      cmd/internal/buildid    1.694s
ok      cmd/internal/cov        1.082s
ok      cmd/internal/dwarf      0.081s
ok      cmd/internal/edit       0.078s
ok      cmd/internal/goobj      0.110s
ok      cmd/internal/moddeps    3.164s
ok      cmd/internal/notsha256  0.101s
ok      cmd/internal/obj        0.757s
ok      cmd/internal/obj/ppc64  0.792s
ok      cmd/internal/obj/riscv  0.201s
ok      cmd/internal/obj/s390x  0.054s
ok      cmd/internal/obj/x86    1.132s
ok      cmd/internal/objabi     0.215s
ok      cmd/internal/pkgpath    0.311s
ok      cmd/internal/pkgpattern 0.089s
ok      cmd/internal/quoted     0.046s
ok      cmd/internal/src        0.044s
ok      cmd/internal/test2json  1.551s
ok      cmd/link        13.448s
ok      cmd/link/internal/benchmark     0.018s
ok      cmd/link/internal/ld    5.684s
ok      cmd/link/internal/loader        0.081s
ok      cmd/nm  4.027s
ok      cmd/objdump     4.722s
ok      cmd/pack        2.489s
ok      cmd/pprof       2.656s
ok      cmd/trace       0.797s
ok      cmd/vet 5.966s
FAIL
go tool dist: Failed: exit status 1

[bcmills]

Like #64257, this does not appear to be WSL-specific.

[gopherbot]

Change https://go.dev/cl/547696 mentions this issue: runtime: fix vet errors for unknown variable name in msan routines

[gopherbot]

Change https://go.dev/cl/547758 mentions this issue: cmd/compile/internal/test: prevent failures in go tool dist test -msan

[mauri870]

@bcmills Do you know if there is a way to detect if msan is enabled, akin to internal/race.Enabled? There are a couple tests that are failing because testing calls are allocating memory, for example this test for slices.Grow

	if n := testing.AllocsPerRun(100, func() { Grow(s2, cap(s2)-len(s2)+1) }); n != 1 {
		errorf := t.Errorf
		if race.Enabled || testenv.OptimizationOff() {
			errorf = t.Logf // this allocates multiple times in race detector mode
		}
		errorf("Grow should allocate once when given insufficient capacity; allocated %v times", n)
	}

I can see runtime.msanenabled, but that is private. I wonder if it would be helpful to have an internal/msan and internal/asan similar to internal/race.

[bcmills]

@mauri870, I see a lot of files in the runtime code with an msan build constraint. Probably you could define a constant for it using build-constrained files?

//go:build msan
…
const msanEnabled = true

//go:build !msan
…
const msanEnabled = false

[mauri870]

@mauri870, I see a lot of files in the runtime code with an msan build constraint. Probably you could define a constant for it using build-constrained files?

As a short term solution, that would work. It is cumbersome to deal with build flags if a specific part of a specific test needs to be skipped, also the failing tests are not only in runtime, I spotted related failures in at least log/slog and slices. I proposed #64611 to make dealing with these cases more straightforward, similar to how we use internal/race.Enabled like in my previous reply.

[zhangfannie]

@mauri870 Just curious, why turning on -msan causes extra allocations? Thanks.

[mauri870]

@zhangfannie It appears to be the same issue we have with -race, t.Errorf allocates memory so there is a hack to use t.Logf as a workaround in some places, e.g

go/src/slices/slices_test.go

Lines 912 to 918 in 2184a39

	if n := testing.AllocsPerRun(100, func() { Grow(s2, cap(s2)-len(s2)+1) }); n != 1 {
	errorf := t.Errorf
	if race.Enabled || testenv.OptimizationOff() {
	errorf = t.Logf // this allocates multiple times in race detector mode
	}
	errorf("Grow should allocate once when given insufficient capacity; allocated %v times", n)
	}

[zhangfannie]

@zhangfannie It appears to be the same issue we have with -race, t.Errorf allocates memory so there is a hack to use t.Logf as a workaround in some places, e.g

Thank you for the explanation. 🙂
But my understanding is that t.Logf is used here to prevent this test from failing, rather than t.Errorf which would cause additional allocations.
What I'm curious about is why does Grow() have extra allocations when -msan or -race is turned on?
@ianlancetaylor
Thank you.

[gopherbot]

Change https://go.dev/cl/622855 mentions this issue: all: skip allocation counting tests with -asan and -msan

[mknyszek]

There are still a few flaky tests here, but they should probably get their own issues, so leaving this closed.

TestNewOSProc0 (wow, sanitizers hate this test)
TestRuntimeLockMetricsAndProfile (timeout?)

[gopherbot]

Change https://go.dev/cl/623176 mentions this issue: runtime: skip TestNewOSProc0 with asan and msan

[gopherbot]

Change https://go.dev/cl/623957 mentions this issue: [release-branch.go1.23] all: skip and fix various tests with -asan and -msan