Merge pull request #1792 from bugsnag/tms/flaky-retries

Allow a single retry on flaky scenarios
bugsnag · Dec 22, 2022 · a70142d · a70142d
2 parents ba7278d + ee29a62
commit a70142d
Show file tree

Hide file tree

Showing 3 changed files with 13 additions and 3 deletions.
diff --git a/features/full_tests/in_foreground.feature b/features/full_tests/in_foreground.feature
@@ -8,4 +8,5 @@ Feature: In foreground field populates correctly
     And I send the app to the background for 5 seconds
     Then I wait to receive an error
     And the error is valid for the error reporting API version "4.0" for the "Android Bugsnag Notifier" notifier
-    And the event "app.inForeground" is false
+    # PLAT-9155 Flaky: the event "app.inForeground" is false
+    And the error is correct for "InForegroundScenario" or I allow a retry
diff --git a/features/full_tests/native_event_tracking.feature b/features/full_tests/native_event_tracking.feature
@@ -17,7 +17,8 @@ Feature: Synchronizing app/device metadata in the native layer
     And I send the app to the background for 5 seconds
     And I wait to receive an error
     Then the error payload contains a completed handled native report
-    And the event "app.inForeground" is false
+    # PLAT-9155 Flaky: the event "app.inForeground" is false
+    And the error is correct for "CXXBackgroundNotifyScenario" or I allow a retry
     And the event "app.durationInForeground" equals 0
     And the event "app.duration" is greater than 0
     And the event "context" string is empty
@@ -43,7 +44,8 @@ Feature: Synchronizing app/device metadata in the native layer
     And I configure Bugsnag for "CXXDelayedCrashScenario"
     And I wait to receive an error
     Then the error payload contains a completed handled native report
-    And the event "app.inForeground" is false
+    # PLAT-9155 Flaky: the event "app.inForeground" is false
+    And the error is correct for "CXXDelayedCrashScenario" or I allow a retry
     And the event "app.duration" is greater than 0
     And the event "context" string is empty
     And the event "unhandled" is true
diff --git a/features/steps/android_steps.rb b/features/steps/android_steps.rb
@@ -261,5 +261,12 @@ def click_if_present(element)
   when 'MultiThreadedStartupScenario'
     Maze.dynamic_retry = true if message == 'You must call Bugsnag.start before any other Bugsnag methods'
     Maze.check.equal 'Scenario complete', message
+  when 'InForegroundScenario', 'CXXBackgroundNotifyScenario', 'CXXDelayedCrashScenario'
+    begin
+      step 'the event "app.inForeground" is false'
+    rescue Test::Unit::AssertionFailedError
+      Maze.dynamic_retry = true
+      raise
+    end
   end
 end