testing: benchmark verbose output changed in 1.14

[robaho]

Using github.com/robaho/fixed and go bench:

In 1.9.7 results are similar to:

goos: darwin
goarch: amd64
pkg: github.com/robaho/fixed
BenchmarkAddFixed-8         	2000000000	         1.26 ns/op
BenchmarkAddDecimal-8       	 5000000	       322 ns/op
BenchmarkAddBigInt-8        	100000000	        14.5 ns/op
BenchmarkAddBigFloat-8      	20000000	        76.5 ns/op
BenchmarkMulFixed-8         	300000000	         4.24 ns/op
BenchmarkMulDecimal-8       	20000000	        69.5 ns/op
BenchmarkMulBigInt-8        	100000000	        17.0 ns/op
BenchmarkMulBigFloat-8      	50000000	        34.4 ns/op
BenchmarkDivFixed-8         	300000000	         4.59 ns/op
BenchmarkDivDecimal-8       	 2000000	       854 ns/op
BenchmarkDivBigInt-8        	30000000	        54.7 ns/op
BenchmarkDivBigFloat-8      	20000000	       118 ns/op

and in 1.14

goos: darwin
goarch: amd64
pkg: github.com/robaho/fixed
BenchmarkAddFixed
BenchmarkAddFixed-8         	1000000000	         0.519 ns/op
BenchmarkAddDecimal
BenchmarkAddDecimal-8       	 5005356	       236 ns/op
BenchmarkAddBigInt
BenchmarkAddBigInt-8        	82658592	        14.9 ns/op
BenchmarkAddBigFloat
BenchmarkAddBigFloat-8      	15501452	        77.2 ns/op
BenchmarkMulFixed
BenchmarkMulFixed-8         	293514025	         4.02 ns/op
BenchmarkMulDecimal
BenchmarkMulDecimal-8       	16031581	        72.6 ns/op
BenchmarkMulBigInt
BenchmarkMulBigInt-8        	70126652	        17.7 ns/op
BenchmarkMulBigFloat
BenchmarkMulBigFloat-8      	33527037	        35.1 ns/op
BenchmarkDivFixed
BenchmarkDivFixed-8         	265002393	         4.53 ns/op

Notice the extra rows without the PROCs count.

[bcmills]

Please fill out the complete issue template. What was the exact command? What does go env report?

[robaho]

Sorry, I thought it was a trivial report. I run the tests via Intellij so I didn't change any parameters - just the GOROOT for the project.

In 1.9.7 the command is

GOROOT=/usr/local/Cellar/[email protected]/1.9.7/libexec #gosetup
GOPATH=/Users/robertengels/go #gosetup
/usr/local/Cellar/[email protected]/1.9.7/libexec/bin/go test -c -o /private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go github.com/robaho/fixed #gosetup
/private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go -test.v -test.bench "^BenchmarkAddFixed|BenchmarkAddDecimal|BenchmarkAddBigInt|BenchmarkAddBigFloat|BenchmarkMulFixed|BenchmarkMulDecimal|BenchmarkMulBigInt|BenchmarkMulBigFloat|BenchmarkDivFixed|BenchmarkDivDecimal|BenchmarkDivBigInt|BenchmarkDivBigFloat|BenchmarkCmpFixed|BenchmarkCmpDecimal|BenchmarkCmpBigInt|BenchmarkCmpBigFloat|BenchmarkStringFixed|BenchmarkStringNFixed|BenchmarkStringDecimal|BenchmarkStringBigInt|BenchmarkStringBigFloat|BenchmarkWriteTo$" -test.run ^$ #gosetup

in 1.14 the command is

GOROOT=/usr/local/Cellar/go/1.14/libexec #gosetup
GOPATH=/Users/robertengels/go #gosetup
/usr/local/Cellar/go/1.14/libexec/bin/go test -c -o /private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go github.com/robaho/fixed #gosetup
/private/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/___gobench_fixed_bench_test_go -test.v -test.bench "^BenchmarkAddFixed|BenchmarkAddDecimal|BenchmarkAddBigInt|BenchmarkAddBigFloat|BenchmarkMulFixed|BenchmarkMulDecimal|BenchmarkMulBigInt|BenchmarkMulBigFloat|BenchmarkDivFixed|BenchmarkDivDecimal|BenchmarkDivBigInt|BenchmarkDivBigFloat|BenchmarkCmpFixed|BenchmarkCmpDecimal|BenchmarkCmpBigInt|BenchmarkCmpBigFloat|BenchmarkStringFixed|BenchmarkStringNFixed|BenchmarkStringDecimal|BenchmarkStringBigInt|BenchmarkStringBigFloat|BenchmarkWriteTo$" -test.run ^$ #gosetup

[robaho]

The output of 'go env' is:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/robertengels/Library/Caches/go-build"
GOENV="/Users/robertengels/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/robertengels/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.14/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.14/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bb/dk8qf0zx6zg7kw3m6dgmj40h0000gn/T/go-build436672849=/tmp/go-build -gno-record-gcc-switches -fno-common"

[robaho]

Interesting too that the 'operations' counts are not whole numbers in 1.14

[mengzhuo]

You have to remove -v

#24929
a813d3c7

[bcmills]

@robaho, it seems like the extra lines here are intended to aid in debugging if one of the benchmarks unexpectedly stalls. Other than being different, are they causing any particular harm?

[robaho]

@mengzhuo I can't - the parameters passed are part of the IntelliJ GoLand/Plugin test runner.

@bcmills not really, but I noticed strange behavior by the test runner in IntelliJ - the progress lines are blank. I have to assume that other CI tools may have problems due to the format change, additionally the non-whole numbers might cause 'regression alerts' in some CI tools.

Just giving a heads-up - not blocking me in any way.

[bcmills]

Thanks for letting us know. Doesn't seem like something we should fix in the Go toolchain at this time.

(FYI @dlsniper; this may be something for you to look into on the IntelliJ side.)

[dlsniper]

but I noticed strange behavior by the test runner in IntelliJ - the progress lines are blank.

@robaho I'd love to have more information on this, but I don't think this is the right place. Please file an issue on our tracker at https://youtrack.jetbrains.com/issues/Go or reach out to us via Gophers Slack in the #goland channel (or me personally using the same handle on Gophers Slack or Twitter).

At the moment I'd rather not change anything because I do like the idea of being able to debug stalling benchmarks, which is the reason this was added in the first place.

As for running without -test.v, we can't really do that as it helps us build the UI around the output from go test.

[robaho]

I filed issue GO-8915 <https://youtrack.jetbrains.com/issue/GO-8915>

…

On Mar 5, 2020, at 12:55 AM, Florin Pățan ***@***.***> wrote: but I noticed strange behavior by the test runner in IntelliJ - the progress lines are blank. @robaho <https://github.com/robaho> I'd love to have more information on this, but I don't think this is the right place. Please file an issue on our tracker at https://youtrack.jetbrains.com/issues/Go <https://youtrack.jetbrains.com/issues/Go> or reach out to us via Gophers Slack in the #goland channel (or me personally using the same handle on Gophers Slack or Twitter). At the moment I'd rather not change anything because I do like the idea of being able to debug stalling benchmarks, which is the reason this was added in the first place. As for running without -test.v, we can't really do that as it helps us build the UI around the output from go test. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#37643?email_source=notifications&email_token=ABF2U4NXTKJKD73EBGYXNY3RF5EERA5CNFSM4LAYESXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN36ERI#issuecomment-595059269>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABF2U4KJPX6Y5FIRDIOD7S3RF5EERANCNFSM4LAYESXA>.