Test Harness Feature Requirements

Requirements For Test Harness

Data collection

• For every test, the harness must be present "pass", "fail", and "partial failure". When "partial failure" is selected, a form with details for partial failure cases will be provide. The structure of this form is provided by the test, it might have different fields for different tests.
• For every test, a text field must be available for copying in the last utterance of the AT after the the operating instructions for the test are preformed.

Testers

• Testers must be able to log in.
• Testers can be approved or unapproved.
• Unapproved testers cannot directly save results to database. There must be a review process.

Test runs

• The test harness must be able to prioritize tests based on which tests need still need to be performed for an AT (with version) and Browser (latest) pair.
• Test results must retain AT version and Browser version numbers.

Definitely out of scope for prototype

Data collection

• Automatic collection of output of AT.
• Users have a concept of "roles", such as "test author" or "reviewer". "reviewers" can mark delete test results or mark test results as inaccurate.