cmd/go: mod tidy leaves go.sum inconsistent after module upgrade

[andig]

What version of Go are you using (go version)?

$ go version
go version go1.17 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/andig/Library/Caches/go-build"
GOENV="/Users/andig/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/andig/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/andig/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/andig/htdocs/evcc/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sv/rs_453y57xj86xsbz3kw1mbc0000gn/T/go-build2957521311=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I've upgraded github.com/avast/retry-go to github.com/avast/retry-go/v3 in my source files, then ran go mod tidy to fix go.mod/sum. Then ran the program

What did you expect to see?

No error

What did you see instead?

core/helper.go:4:2: missing go.sum entry needed to verify package github.com/avast/retry-go/v3 (imported by github.com/andig/evcc/core) is provided by exactly one module; to add:
	go get github.com/andig/evcc/core

Steps to reproduce:

git checkout https://github.com/andig/evcc && cd evcc
git reset b941caf5ea2afd72803f715a4892374dc14748f6 # this is the upgrade commit
git checkout go.* # reset those to before go mod tidy
go mod tidy
go run main.go

[seankhliao]

Seems to work with either a go 1.17 directive in go.mod or -compat=1.17

cc @bcmills @jayconrod @matloob

[jayconrod]

Reproduced and confirmed.

After go mod tidy runs, module github.com/avast/retry-go is still indirectly required, which means it still ends up in the build list. Because both github.com/avast/retry-go and github.com/avast/retry-go/v3 are in the build list, when package github.com/avast/retry-go/v3 is loaded, the go command needs the sums for both modules to confirm the package only exists in one. But go mod tidy incorrectly deleted that sum.

[jayconrod]

Looked into this further.

go mod tidy deletes the requirement on github.com/avast/retry-go v3.0.0+incompatible and the corresponding sums. However, github.com/avast/retry-go is still indirectly required at v2.6.0+incompatible so that module is effectively downgraded. That's fine because it doesn't provide any packages needed to build packages in the main module and their tests.

Tidy still needs to fetch the new zip file for v2.6.0+incompatible and add the sum for that. Amusingly a second pass of go mod tidy fixes the problem.

This also reproduces with 1.16.7, so it's not a regression in 1.17.

Test case is here if anyone has a chance to look at this sooner.

go list -m rsc.io/quote
stdout '^rsc.io/quote v1.5.2$'

go mod tidy
go list -m rsc.io/quote
stdout '^rsc.io/quote v1.5.1$'
go build -n .

-- go.mod --
module use

go 1.16

require (
	old-indirect v0.0.0
	rsc.io/quote v1.5.2
)

replace old-indirect v0.0.0 => ./old-indirect
-- go.sum --
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:pvCbr/wm8HzDD3fVywevekufpn6tCGPY3spdHeZJEsw=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.1/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/quote v1.5.2 h1:3fEykkD9k7lYzXqCYrwGAf7iNhbk4yCjHmKBN9td4L0=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0 h1:HLGR/BgEtI3r0uymSP/nl2uPLsUnNJX8toRyhfpBTII=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
-- use.go --
package use

import (
	_ "old-indirect/empty"

	_ "rsc.io/quote/v3"
)
-- old-indirect/empty/empty.go --
package empty
-- old-indirect/go.mod --
module old-indirect

go 1.16

require rsc.io/quote v1.5.1
-- old-indirect/go.sum --
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

[gopherbot]

Change https://golang.org/cl/344572 mentions this issue: cmd/go/internal/modload: prevent tidy downgrading disambiguating modules

[gopherbot]

Change https://golang.org/cl/378577 mentions this issue: doc/go1.18: add a release note for 'go mod tidy' checksum changes