Fix SentryTests.SentrySessionTrackerTests Flaky tests

[brustolin]

Description

This tests are really flaky right now.

[kevinrenskers]

It happens especially often on tvOS it seems, and rerunning the failing tests always fixes it (sometimes you need to do it more than once though). Why is this so unreliable? How can we improve this? Why are tests not automatically retried? Is GitHub really the best CI platform for this?

It's usually 57 failures. This happens both in "Unit tvOS - Xcode 13.2.1 - OS latest" and "Unit tvOS - Xcode 12.5.1 - OS latest".

Examples:

https://github.com/getsentry/sentry-cocoa/runs/7766682459?check_suite_focus=true
https://github.com/getsentry/sentry-cocoa/runs/7804389201?check_suite_focus=true
https://github.com/getsentry/sentry-cocoa/runs/7745310447?check_suite_focus=true

Some of them have this error when the tests start, but not all of them: Failed to compute path to baseline file during test run spec construction.

Some of the failures are confusing and the message not helpful: XCTAssertEqual failed: ("SentrySessionStatus") is not equal to ("SentrySessionStatus")

[kevinrenskers]

Running them repeatedly (100 times) locally, for both iOS and tvOS, and everything is fine. I really do wonder why it's so bad on GitHub's CI.

[armcknight]

OK well, now we know which statuses we're dealing with for that one confusing message: https://github.com/getsentry/sentry-cocoa/runs/7904857914?check_suite_focus=true#step:9:25

Expected session status of 0 but got 1 (which correspond to "ok" and "exited", respectively)

[armcknight]

For future reference, these are currently the flaky test cases I'm seeing in SentrySessionTrackerTests:

testAppNotRunning_LaunchBackgroundTask_UserOpensApp
testAppRunningInForeground_LaunchFromBackground_Terminate
testAppRunning_LaunchBackgroundTaskImmidiately_UserResumesApp
testAppRunning_LaunchBackgroundTask_UserOpensApp
testCrashInBackground_LaunchInForeground
testCrashInForeground_LaunchInForeground
testForegroundWithError
testForeground_Background_Foreground_NoSessionToEnd
testForeground_Background_Terminate_LaunchAgain
testForeground_Background_TrackingIntervalNotReached
testForeground_Background_TrackingIntervalReached
testKillAppWithoutNotificationsAndNoCrash_EndsWithAbnormalSession

And also SentrySystemEventBreadcrumbsTest.testTimezoneChangeNotificationBreadcrumb() but that is probably a different problem than all the ones in SentrySessionTrackerTests. See #1930 for more discussion on that one.

[kevinrenskers]

Running them locally in Xcode doesn't cause any failures, so I also tried to run them a bunch of times using the same command line invocation as we use on CI:

env NSUnbufferedIO=YES xcodebuild -workspace Sentry.xcworkspace -scheme Sentry -configuration "Test" GCC_GENERATE_TEST_COVERAGE_FILES=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES -destination "platform=iOS Simulator,OS=latest,name=iPhone 8" test

(I did first create an iPhone 8 simulator in Xcode 14)

And yeah.. still all fine, no failures other than #2164.

I also tried with the tvOS simulator, since those fail frequently on CI:

env NSUnbufferedIO=YES xcodebuild -workspace Sentry.xcworkspace -scheme Sentry -configuration "Test" GCC_GENERATE_TEST_COVERAGE_FILES=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES -destination "platform=tvOS Simulator,OS=latest,name=Apple TV" test

But of course that is tvOS 16 using Xcode 14, so not quite the same as our CI runs. But I have run these tests plenty of time in Xcode 13 using the older tvOS simulator, and never any problems.

So the big question: why do they fail on CI, and pass locally? What can we do about it? Since these tests do seem to pass in Xcode 14 on CI, maybe we disable this test class for older Xcodes?