UITests/DeviceTests TODOS

[rmarinho]

Description

We are trying to get UI testing going here's some of what's left to do:

High priority

• For WinUI Controls Device Tests add more logging: (optionally) log which tests are actually being ran during the test run so we have a better idea of what actually runs
• For WinUI Controls Device Tests try and make it better but that will be more long term
  • Follow up with WinUI team to track limitations and bugs on their side
• Move FlyoutNavigationBetweenItemsWithNavigationStacks to Appium Tests
• [Windows] Review currently failing Windows tests #15530
• WebKit crashes https://dev.azure.com/xamarin/public/_build/results?buildId=92831&view=logs&jobId=55a4ab5f-e3bf-5649-fe78-d5500c5b0aed&j=55a4ab5f-e3bf-5649-fe78-d5500c5b0aed&t=ec3e639b-355c-5da8-526e-ca4a339c55a0
• Review mechanisms for allowing tests to efficiently run in isolation. Currently if you have multiple tests cases you're required to manually reset for each test case. We've used methods like "push test and then pop at bottom". With Legacy tests we used the BackDoor feature to reload the mainpage between every run. @BretJohnson is working on remote calls that we could use to reset, we could reset the entire appium test and pass in capabilities where to start?

Pending

• Use the Integration project to setup and start emulator
• Add mechanism to only run tests on certain devices, not all
• resolve issue with installing node
• [UITest] Fix naming of test file on CI so we don't overwrite the files on attempts #16506
• [UITests] Fix naming of UITests so we get more information than '_IsEnabled` etc... #16505
• When running one test on a given platform it runs the same test on all platforms
• fix iOS.cake script to detect if XCODE_PATH Exists and if it doesn't just use XCODE.App

  maui/eng/devices/ios.cake

  Line 169 in 412eb05

  var XCODE_PATH = "/Applications/Xcode_14.3.0.app";

  currently when running device tests if you don't have the XCODE path set with the version suffix it won't run
• Matt's extraction of DeviceTestRunners https://github.com/mattleibow/DeviceRunners
• Manuel's xHarness work
• Frequent Failures with trying to go back to test button
  • Test runner no longer completely fails but we would still like to figure out why these failures happen. We take a screen shot and log when this happens
• Convert UITests to xunit so we are on the same testing stack everywhere
• Add very basic sample of using Appium with the MAUI templated app. Setting up MAUI with Appium has a few small quirks, for example you have to set noREset to true on Android for it to work. Searching by AutomationId is different on Android vs iOS/catalyst. So the sample would be the most bare bones examples of appium working where it can click the button on the sample app. We won't wire up any of the server starting code we have in our tests. Users will need to install appium and start the server themselves.
• Flesh out visual testing with Border components. Currently the visual testing features are very much tied to the machine. For example, we have to generate these on CI to get a testable image even though those won't now pass locally.

Low priority

• Restructure directories
• Update / finish doc
• Make sure we are able to use the local and global workloads to run the tests

FINISHED

• Fix iOS Visual Runner so that all tests pass consistently Fix iOS Visual Runner so that all tests pass consistently #18378
• Fix this test for iOS17 Ignore visual tests on iPhone 15 devices #18407
• If legacy tests aren't running and the results are empty then fail the run [UITests] Legacy iOS tests currently not producing results on main #17945
• [regression/8.0.0] [Windows] Fix CollectionView tests that have regressed #15529
• Make sure project structure and slnf are all updated
• Check appium log file locations when running locally or on CI
• Add more debug info when build /deploying /running
• Try run Xamarin.UITest with iOS and Android on CI
• Build app from the test project
• Fixing CI flakiness Compatibility.
  • Can't find mono
• Fixing CI flakiness Appium
  • ps1 script is unreliable with installing appium.
  • installing the drivers might be the issue?
• Fix WinUI Controls Device Tests (Enable WinUI Controls Test Pipeline #17221)
  • here are the Categories
  • make it restartable
  • Split out the tests that don't need new windows and let those ones run in current state
  • Fix WinUI Controls Device tests disabled tests, see Enable WinUI Controls Test Pipeline #17221
• Fix Catalyst Device Tests Controls (Enable Mac Catalyst Controls Device Tests in CI #17229)
  • Fix Catalyst Controls Device tests disabled tests, see Enable Mac Catalyst Controls Device Tests in CI #17229
• Fix Catalyst Device Tests Core (Enable Mac Catalyst Core Device Tests in CI #17232)
  • Fix Catalyst Core Device tests disabled tests, see Enable Mac Catalyst Core Device Tests in CI #17232
• Reenable WhenQueryingCarouselItemsInViewThenSingleItemIsRetrieved Ignore failures from newly added UITests temporarily #17003

CUT

• Try run the new test with appium against the Control Gallery

[ghost]

We've added this issue to our backlog, and we will work to address it as time and resources allow. If you have any additional information or questions about this issue, please leave a comment. For additional info about issue management, please read our Triage Process.