runtime: "morestack on g0" in TestSegv on darwin-amd64 builders #39457

bcmills · 2020-06-08T19:53:42Z

2020-06-08T17:59:37-2603d9a/darwin-amd64-race

--- FAIL: TestSegv (0.00s)
    --- FAIL: TestSegv/Segv (0.02s)
        crash_test.go:105: /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/go-build172134279/testprogcgo.exe SegvInCgo exit status: exit status 2
        crash_cgo_test.go:569: fatal: morestack on g0
            SIGTRAP: trace trap
            PC=0x406b702 m=0 sigcode=1
            
            goroutine 0 [idle]:
            runtime.abort()
            	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/asm_amd64.s:860 +0x2
            runtime.morestack()
            	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/asm_amd64.s:416 +0x25
            
            goroutine 19 [syscall]:
            runtime.cgocall(0x4123600, 0xc00003a7c0, 0x4123600)
            	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/cgocall.go:133 +0x5b fp=0xc00003a790 sp=0xc00003a758 pc=0x400503b
            main._Cfunc_nop()
            	_cgo_gotypes.go:329 +0x45 fp=0xc00003a7c0 sp=0xc00003a790 pc=0x411a2a5
            main.SegvInCgo.func1(0xc00008e120)
            	/private/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/testdata/testprogcgo/segv.go:46 +0x30 fp=0xc00003a7d8 sp=0xc00003a7c0 pc=0x41224b0
            runtime.goexit()
            	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00003a7e0 sp=0xc00003a7d8 pc=0x406b8e1
            created by main.SegvInCgo
            	/private/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/testdata/testprogcgo/segv.go:43 +0x5c
            
            goroutine 1 [sleep]:
            time.Sleep(0x3b9aca00)
            	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/time.go:188 +0xbf
            main.SegvInCgo()
            	/private/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/testdata/testprogcgo/segv.go:55 +0x9c
            main.main()
            	/private/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/go/src/runtime/testdata/testprogcgo/main.go:34 +0x1da
            
            rax    0x17
            rbx    0xc00003a710
            rcx    0x4265d40
            rdx    0x0
            rdi    0x2
            rsi    0xc00003a6b0
            rbp    0xc00003a780
            rsp    0xc00003a738
            r8     0x4265d40
            r9     0x0
            r10    0xc00003a710
            r11    0x202
            r12    0xf1
            r13    0x0
            r14    0x418de44
            r15    0x0
            rip    0x406b702
            rflags 0x202
            cs     0x2b
            fs     0x0
            gs     0x0
            
        crash_cgo_test.go:571: expected crash from signal
FAIL
FAIL	runtime	69.144s

CC @aclements @ianlancetaylor @cherrymui

The text was updated successfully, but these errors were encountered:

cherrymui · 2020-06-17T00:37:15Z

On darwin/amd64, to work around a kernel issue we rewrite SI_USER SIGEGV to kernel-generated: https://tip.golang.org/src/runtime/signal_darwin_amd64.go#L72 . So, in this case, an actual user-sent SIGSEGV will be treated as kernel-generated signal, and cause it to inject a sigpanic. If the signal lands at a bad time, e.g. we're right in the middle of a stack switch, where the g and the stack don't match, bad things will happen.

I'm not sure what the best solution is. A few possibilities:

do nothing (maybe skip/relax the test). It isn't too bad in that it will crash the program anyway (sigpanic will throw for this particular bad address), unless PanicOnFault is set.
remove the workaround (at least the sigcode part, we could still change the faulting address). A malformed address will be treated as user-sent SIGSEGV, which will crash the program now. PanicOnFault is still a problem.

Not sure what to do with PanicOnFault. Due to the kernel issue, it seems we cannot distinguish malformed address vs. user-sent SIGSEGV. We have to make both recoverable or non-recoverable...

(The workaround was added for OS X 10.9. The kernel issue seems still there for macOS 10.15...)

cherrymui · 2020-06-17T00:47:01Z

Another possibility: when switching from user stack to system stack (e.g. in systemstack, asmcgocall, etc.), we always do (1) set user g's g.throwsplit to true, (2) change SP, (3) change the g register to g0. And do it in the opposite order when switching back. This might solve the immediate SIGSEGV-landing-in-stack-switch problem. Not sure if there is any other problem. Seems pretty complicated, though.

bcmills · 2020-08-31T18:33:25Z

2020-08-18T04:22:33-6133883/darwin-amd64-10_12
2020-08-14T23:04:16-5a18e0b/darwin-amd64-10_15
2020-06-16T19:29:32-844bf11/darwin-amd64-race

bcmills · 2020-08-31T18:41:57Z

Hmm... Why do we ignore user-generated SIGSEGV signals in the first place? I explicitly sent a program SIGSEGV on the command line, I would generally expect to get a core dump (since that is the SIG_DFL behavior of the signal to begin with).

ianlancetaylor · 2020-08-31T18:50:23Z

We don't ignore user-generated SIGSEGV signals. That's the point of the test. I'm not sure what you are saying here.

The test failure logs suggest that the problem is that we somehow think that we are out of stack space while handling a signal. I'm not sure how that could happen.

cherrymui · 2020-08-31T20:16:20Z

In my experience, "morestack on g0" is usually not we are actually running out of stack space on g0, but somehow the SP and and the G (and so the stack bounds) don't match. My comment above mentioned some possibilities, e.g. signal lands right in the middle of a stack switch.

As @ianlancetaylor said, we don't ignore user-generated SIGSEGV (we did it in the past, but not now). The specialness of darwin is that we treat user-generated SIGSEGV (which should crash the runtime) as kernel-generated (which causes a panic), due to a kernel bug ( https://tip.golang.org/src/runtime/signal_darwin_amd64.go#L72 ). Because of that, we inject a sigpanic call, instead of just throw, and somewhere down the panic path there are non-nosplit functions that check stack bounds. If the G and stack bounds don't match, it could crash like this.

bcmills · 2021-02-22T15:02:48Z

2021-02-21T02:25:41-0f66fb7/darwin-amd64-10_14
2020-12-14T22:28:10-48906a6/darwin-amd64-10_14
2020-10-27T03:11:45-a19cf51/darwin-amd64-10_15
2020-10-17T00:32:32-11cfb48/darwin-amd64-10_15

bcmills · 2021-06-08T01:14:39Z

2021-06-07T15:16:23-e3176bb/darwin-amd64-noregabi
2021-04-30T05:18:37-9c72078/darwin-amd64-10_15
2021-04-14T21:00:20-23f8c20/darwin-amd64-11_0
2021-04-06T03:31:38-d8306ee/darwin-amd64-10_15
2021-04-02T21:57:58-f25d78f/darwin-amd64-10_15

bcmills · 2022-02-02T15:14:13Z

greplogs --dashboard -md -l -e 'FAIL: TestSegv .*(?:\n .*)*fatal: morestack on g0' --since=2021-06-08

2022-02-01T21:46:52-36b81ac/darwin-amd64-10_15
2022-01-10T20:53:01-55d10ac/darwin-amd64-12_0
2021-11-09T18:23:16-55e6e82/darwin-amd64-10_15
2021-11-05T22:28:56-87a3fc5/darwin-amd64-10_14
2021-10-27T16:39:27-4f73fd0/darwin-amd64-race
2021-09-17T21:29:32-3fa7dbe/darwin-amd64-10_15
2021-09-12T16:46:58-ad97d20/darwin-amd64-10_15
2021-09-09T12:57:26-66f0d35/darwin-amd64-10_14
2021-09-02T19:36:22-a8aa6cf/darwin-amd64-noregabi
2021-08-27T20:43:31-72c003e/darwin-amd64-noregabi
2021-08-19T00:26:22-c85695a/darwin-amd64-11_0
2021-08-18T08:26:44-8b471db/darwin-amd64-11_0
2021-08-15T19:08:56-6ed9463/darwin-amd64-race
2021-06-30T16:44:19-0fa3265/darwin-amd64-race
2021-06-17T04:32:50-97cee43/darwin-amd64-race

bcmills · 2022-02-02T15:16:45Z

darwin/amd64 is a first class port, and this test has been failing intermittently on the builder for over a year and a half.

If the behavior covered by this test is important then we really ought to find a solution for it; otherwise, the test should be skipped to reduce noise on the builders.

gopherbot · 2022-02-02T15:24:32Z

Change https://golang.org/cl/382395 mentions this issue: runtime: skip TestSegv failures with "morestack on g0" on darwin/amd64

This failure mode has been present since at least 2020-06-08. We have enough information to diagnose it, and further failures don't seem to be adding any new information at this point: they can only add noise, both on the Go project's builders and in users' own modules (for example, when run as part of 'go test all'). For #39457 Change-Id: I2379631da0c8af69598fa61c0cc5ac0ea6ba8267 Reviewed-on: https://go-review.googlesource.com/c/go/+/382395 Trust: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Cherry Mui <[email protected]>

evanw · 2022-02-18T19:55:00Z

I have a user report about this error on a Windows machine with esbuild, which is written in Go but doesn't use cgo: evanw/esbuild#2031. I searched and found this issue and that report seemed potentially related, so I'm posting about it here in case it helps.

cherrymui · 2022-02-18T20:57:22Z

@evanw that looks like a different issue. Could you open a new issue? Thanks.

This issue is very specific to darwin (macOS) when a program receives an external SIGSEGV signal (e.g. by kill command or syscall).

bcmills added OS-Darwin NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 8, 2020

bcmills added this to the Backlog milestone Jun 8, 2020

golang deleted a comment Jun 10, 2020

royari mentioned this issue Jul 17, 2020

macOS kapitan release proposal kapicorp/kapitan#568

Closed

AskAlexSharov mentioned this issue Aug 9, 2020

stage3: morestack on g0, SIGTRAP erigontech/erigon#892

Closed

bcmills changed the title ~~runtime: "morestack on g0" in TestSegv on darwin-amd64-race builder~~ runtime: "morestack on g0" in TestSegv on darwin-amd64 builders Aug 31, 2020

evanw mentioned this issue Feb 18, 2022

Intermittent crashes during build evanw/esbuild#2031

Closed

evanw mentioned this issue Feb 18, 2022

runtime: esbuild sometimes crashes with fatal: morestack on g0 on 64-bit Windows #51266

Closed

bcmills mentioned this issue Feb 24, 2022

runtime: TestSegv/SegvInCgo failure with unexpected return pc for runtime.casgstatus on linux/arm #51353

Closed

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022

mknyszek added this to Go Compiler / Runtime Jul 7, 2022

mknyszek moved this to Triage Backlog in Go Compiler / Runtime Jul 15, 2022

bcmills mentioned this issue May 3, 2023

runtime: TestSegv/SegvInCgo timeout on ios-arm64-corellium #59947

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: "morestack on g0" in TestSegv on darwin-amd64 builders #39457

runtime: "morestack on g0" in TestSegv on darwin-amd64 builders #39457

bcmills commented Jun 8, 2020 •

edited

Loading

cherrymui commented Jun 17, 2020

cherrymui commented Jun 17, 2020

bcmills commented Aug 31, 2020

bcmills commented Aug 31, 2020

ianlancetaylor commented Aug 31, 2020

cherrymui commented Aug 31, 2020

bcmills commented Feb 22, 2021

bcmills commented Jun 8, 2021

bcmills commented Feb 2, 2022

bcmills commented Feb 2, 2022

gopherbot commented Feb 2, 2022

evanw commented Feb 18, 2022

cherrymui commented Feb 18, 2022

runtime: "morestack on g0" in TestSegv on darwin-amd64 builders #39457

runtime: "morestack on g0" in TestSegv on darwin-amd64 builders #39457

Comments

bcmills commented Jun 8, 2020 • edited Loading

cherrymui commented Jun 17, 2020

cherrymui commented Jun 17, 2020

bcmills commented Aug 31, 2020

bcmills commented Aug 31, 2020

ianlancetaylor commented Aug 31, 2020

cherrymui commented Aug 31, 2020

bcmills commented Feb 22, 2021

bcmills commented Jun 8, 2021

bcmills commented Feb 2, 2022

bcmills commented Feb 2, 2022

gopherbot commented Feb 2, 2022

evanw commented Feb 18, 2022

cherrymui commented Feb 18, 2022

bcmills commented Jun 8, 2020 •

edited

Loading