test: process to skip tests

[armcknight]

After trialing a process to add a _disabled suffix to skipped tests, we will forego that in favor of solely relying on the Xcode scheme to manage the skipped tests. Here are a couple reasons why:

1. it's already not being done in some cases. Recent commits have not added the suffix, and comparing the list of skipped tests in the scheme vs tests with a _disabled suffix yields a bunch of differences:

❯ diff -y ~/Desktop/from_names.txt ~/Desktop/from_scheme.txt
test_DataConsistency_readUrl_disabled			      |	testStartUpCrash_CallsFlush_disabled
							      >	testCrashReportCount1_disabled
test_DataConsistency_readPath_disabled				test_DataConsistency_readPath_disabled
							      >	test_DataConsistency_readUrl_disabled
							      >	testSerializeWithUnderlyingNSException_disabled
testGetRequest_SpanCreatedAndBaggageHeaderAdded_disabled	testGetRequest_SpanCreatedAndBaggageHeaderAdded_disabled
testActOnSubclassesOfViewController_disabled		      <
test_attachScreenshot_disabled				      <
testStartUpCrash_CallsFlush_disabled			      <
testSendSessions_disabled				      <
test_attachViewHierarchy_disabled			      <
testConcurrentSpansWithTimeout_disabled				testConcurrentSpansWithTimeout_disabled
testProfileTimeoutTimer_disabled				testProfileTimeoutTimer_disabled
testSerializeWithUnderlyingNSException_disabled		      |	testSendSessions_disabled
testCrashReportCount1_disabled				      |	testAsyncStacktraces_disabled
testAsyncStacktraces_disabled				      |	testActOnSubclassesOfViewController_disabled⏎

I am leaving the tests that were named with a _disabled suffix but not actually skipped in test runs as not-skipped, since they are apparently running today without issue.

2. there are other tests with _disabled suffixes where "disabled" doesn't mean the test is skipped, but rather that they're testing the SDK with some of its features turned off.

3. when fixing the test case, unless you do a specific order of steps, the xcscheme file accumulates cruft. here's how: if you rename the test to remove the suffix, the xcscheme file still contains the skipped name with the suffix. xcode isn't smart enough to know that we're renaming the test to reenable it. So then we'll have a test that does run, and a skipped test name in the scheme file that doesn't exist. you have to first reenable the test to remove it from the skipped test list in the scheme, and then rename it.

Opened a few new issues for tests that were skipped with no associated issue:

• Fix flaky test_applicationWithScenes_noWindow #2607
• Fix flaky test_applicationWithScenes #2606
• Fix flaky testGetRequest_SpanCreatedAndBaggageHeaderAdded #2605
• Fix flaky testStartUpCrash_CallsFlush1 #2604

#skip-changelog

[codecov]

Codecov Report

Merging #2603 (195f560) into main (23e1220) will increase coverage by 0.03%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2603      +/-   ##
==========================================
+ Coverage   79.67%   79.71%   +0.03%     
==========================================
  Files         244      243       -1     
  Lines       22618    22630      +12     
  Branches    10008    10018      +10     
==========================================
+ Hits        18022    18040      +18     
+ Misses       4145     4140       -5     
+ Partials      451      450       -1     

Impacted Files	Coverage Δ
Sources/SentryCrash/Recording/Tools/SentryHook.c	86.02% <0.00%> (-1.48%)	⬇️
Sources/Sentry/include/SentryHexAddressFormatter.h
Sources/Sentry/SentryNetworkTracker.m	94.53% <0.00%> (+0.27%)	⬆️
Sources/Sentry/SentryFileManager.m	95.66% <0.00%> (+0.61%)	⬆️
Sources/Sentry/NSDictionary+SentrySanitize.m	100.00% <0.00%> (+4.00%)	⬆️
Sources/Sentry/SentryRequestOperation.m	78.37% <0.00%> (+10.81%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23e1220...195f560. Read the comment docs.

[philipphofmann]

Thanks a lot @armcknight 🥇