Test failure: baseservices/threading/generics/WaitCallback/thread17/thread17.sh

[v-haren]

failed in job: runtime-coreclr gcstress-extra 20200524.1

Error message

Assert failure(PID 11756 [0x00002dec], Thread: 11976 [0x2ec8]): IsExecutingOnAltStack() || (curSP <= m_pFrame && m_pFrame < m_CacheStackBase) || m_pFrame == (Frame*) -1
 File: /__w/1/s/src/coreclr/src/vm/threads.h Line: 1859
 Image: /root/helix/work/correlation/corerun

/root/helix/work/workitem/baseservices/threading/generics/WaitCallback/thread17/thread17.sh: line 317: 11756 Aborted $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code: 1
Raw output file: /root/helix/work/workitem/baseservices/threading/Reports/baseservices.threading/generics/WaitCallback/thread17/thread17.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun thread17.dll ''
Writing minidump to file /home/helixbot/dotnetbuild/dumps/coredump.11756.dmp
Written 67555328 bytes (16493 pages) to core file
Dump successfully written
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/root/helix/work/correlation
> /root/helix/work/workitem/baseservices/threading/generics/WaitCallback/thread17/thread17.sh
Expected: True
Actual: False


Stack trace
   at baseservices_threading._generics_WaitCallback_thread17_thread17_._generics_WaitCallback_thread17_thread17_sh() in /__w/1/s/artifacts/tests/coreclr/Linux.arm.Checked/TestWrappers/baseservices.threading/baseservices.threading.XUnitWrapper.cs:line 12265

category:correctness
theme:gc-stress
skill-level:expert
cost:medium

[BruceForstall]

Failure mode: CoreCLR Linux arm Checked gcstress0xc_zapdisable_heapverify1

[AndyAyersMS]

Seems likely to be the same issue as in #36636, so will assign to myself.

[v-haren]

failed again in job: runtime-coreclr gcstress-extra 20200607.1

failed test: baseservices/threading/generics/WaitCallback/thread28/thread28.sh

Error message

Assert failure(PID 2658 [0x00000a62], Thread: 2915 [0x0b63]): IsExecutingOnAltStack() || (curSP <= m_pFrame && m_pFrame < m_CacheStackBase) || m_pFrame == (Frame*) -1
 File: /__w/1/s/src/coreclr/src/vm/threads.h Line: 1859
 Image: /root/helix/work/correlation/corerun

/root/helix/work/workitem/baseservices/threading/generics/WaitCallback/thread28/thread28.sh: line 317: 2658 Aborted $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code: 1
Raw output file: /root/helix/work/workitem/baseservices/threading/Reports/baseservices.threading/generics/WaitCallback/thread28/thread28.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun thread28.dll ''
Writing minidump to file /home/helixbot/dotnetbuild/dumps/coredump.2658.dmp
Written 67571712 bytes (16497 pages) to core file
Dump successfully written
Expected: 100
Actual: 134
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/root/helix/work/correlation
> /root/helix/work/workitem/baseservices/threading/generics/WaitCallback/thread28/thread28.sh
Expected: True
Actual: False


Stack trace
   at baseservices_threading._generics_WaitCallback_thread28_thread28_._generics_WaitCallback_thread28_thread28_sh() in /__w/1/s/artifacts/tests/coreclr/Linux.arm.Checked/TestWrappers/baseservices.threading/baseservices.threading.XUnitWrapper.cs:line 13332

[AndyAyersMS]

Now seems to be failing most consistently on arm32, so will see if I can get a repro there.

[AndyAyersMS]

Suspect the arm32 failures are the jit-stress induced profiler leave hook not properly maintaining GC liveness of the return value.

@CarolEidt I am thinking #37679 may be a contributing factor, as the return value is temporarily held in r2 around the profiler callback. Failing method is String.Format.

000086  4602           mov     r2, r0
000088  F245 4011      movw    r0, 0x5411       +r2
00008C  F6CE 20E3      movt    r0, 0xeae3       -r0
000090  F245 4C11      movw    r12, 0x5411
000094  F6CE 2CE3      movt    r12, 0xeae3
000098  47E0           blx     r12              // CORINFO_HELP_PROF_FCN_LEAVE
00009A  4610           mov     r0, r2           -r2
00009C  F245 6C78      movw    r12, 0x5678      +r0  ------------------ fails here
0000A0  F2C1 2C34      movt    r12, 0x1234      
0000A4  F8DD E024      ldr     lr, [sp+0x24]    // [V06 GsCookie]
0000A8  45F4           cmp     r12, lr
0000AA  D004           beq     SHORT G_M58996_IG47
0000AC  F24E 7C1D      movw    r12, 0xe71d
0000B0  F2CF 4CCA      movt    r12, 0xf4ca
0000B4  47E0           blx     r12              // CORINFO_HELP_FAIL_FAST

G_M58996_IG47:
0000B6  B00A           add     sp, 40

Runtime complains r0 is an invalid GC ref at that point.

cc @BruceForstall

[v-haren]

also failed in job: runtime-coreclr jitstress 20200704.1

failed test: baseservices/threading/paramthreadstart/ThreadStartOperations_1/ThreadStartOperations_1.sh

Error message

Assert failure(PID 1066 [0x0000042a], Thread: 1088 [0x0440]): IsExecutingOnAltStack() || (curSP <= m_pFrame && m_pFrame < m_CacheStackBase) || m_pFrame == (Frame*) -1
 File: /__w/1/s/src/coreclr/src/vm/threads.h Line: 1859
 Image: /root/helix/work/correlation/corerun

../ThreadStartOperations/ThreadStartOperations.sh: line 318: 1066 Aborted $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code: 1
Raw output file: /root/helix/work/workitem/baseservices/threading/Reports/baseservices.threading/paramthreadstart/ThreadStartOperations_1/ThreadStartOperations_1.output.txt
Raw output:
BEGIN EXECUTION
export CDPATH=/root/helix/work/workitem/baseservices/threading/paramthreadstart/ThreadStartOperations_1
/usr/bin/env bash ../ThreadStartOperations/ThreadStartOperations.sh 'min'
BEGIN EXECUTION
/root/helix/work/correlation/corerun ThreadStartOperations.dll 'min'
Writing minidump to file /home/helixbot/dotnetbuild/dumps/coredump.1066.dmp
Written 77410304 bytes (18899 pages) to core file
Dump successfully written
Expected: 100
Actual: 134
END EXECUTION - FAILED
Expected: 0
Actual: 1
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/root/helix/work/correlation
> /root/helix/work/workitem/baseservices/threading/paramthreadstart/ThreadStartOperations_1/ThreadStartOperations_1.sh
Expected: True
Actual: False


Stack trace
   at baseservices_threading._paramthreadstart_ThreadStartOperations_1_ThreadStartOperations_1_._paramthreadstart_ThreadStartOperations_1_ThreadStartOperations_1_sh() in /__w/1/s/artifacts/tests/coreclr/Linux.arm.Checked/TestWrappers/baseservices.threading/baseservices.threading.XUnitWrapper.cs:line 27882

[AndyAyersMS]

These haven't failed in the past 3 weekend runs, so closing.