Skip to content

Workstation Acceptance Tests

Jump to bottom
deeplow edited this page Jul 3, 2024 · 53 revisions

SecureDrop Workstation test scenarios

Some scenarios require a supported printer. We're tracking compatible printers available to team members here.

Also see https://drive.google.com/drive/u/0/folders/1lyk2V15e7amI9OxvgUK_YhYIUVNDWJo0 for different types of file submissions we'll want to test for printing and display (mostly pulled from https://file-examples.com/).

Possible way to break up testing:

Creative/exploratory testing: Beyond the test plan, it's helpful to be a bit creative in a way that simulates a normal user - with emphasis on the last part. (Clicking a button a few times to see if there are any adverse effects on the UI - helpful! Using an automated framework to spam the GUI with hundreds of clicks per second to see what happens - less helpful!). Document any exploratory testing, impressions and observations, and ideally, include steps for others to reproduce your findings.

Some examples:

  • Qubes-related interactions: How's updating? How's attaching, removing, and interacting with peripheral devices? Do our docs clearly state how to recover from scenarios such as failed network connection, failed updates, etc? What happens if you disconnect the network in the middle of an update, use toxiproxy to simulate slow downloads, etc?
  • Large file issues: Can large files be downloaded and/or moved between VMs? Can conditions be introduced that would make some large file downloads problematic (eg: a several-megabyte file)?
  • Large volume of submissions: What happens when there are lots of submissions on the server?
  • Impatient/hurried user: What are your observations if you approach your testing session from the point of view of being in a hurry, short on time, rapidly clicking, etc

Some of the feedback may be bug reports and/or UX observations that aren't directly bug reports.

Internationalization (reference)

At least one tester should check that:

  • Under the LANG=en_XA pseudolocale, all GUI strings like foo are correctly wrapped like [!foo!]. (Any string not so wrapped is missing its gettext() wrapper.)
  • Generate a conversation transcript and export it. Inspect to make sure it displays correctly in the specified locale.

Qubes scenarios

Some of these may be a bit time-consuming, so we typically have a subset of devs test these specific scenarios. In these scenarios, we try to validate the behavior of different components of the system. There's a separate section with detail on Client functionality specifically.

The Qubes scenarios are tracked in the requirements tracing document. This is an effort to ensure most (if not all system functionality) is covered by QA / automated tests. This temporarily lives in a google doc, but the goal to move it to the repository itself and have each requirement mapped to its implementation and respective test with the help of a tool like OpenFastTrace.

Client scenarios

Some client tests do not require being run in a SecureDrop Workstation environment. The list of scenarios below is a complete list for full acceptance testing, which should be done in the supported target Qubes environment.

Scenario: Online mode

Prerequisites:

  • server is available and contains source test data
  • access to sd-gpg keyring has not been previously granted
  • ~/.securedrop_client/data in sd-app is empty, and ~/.securedrop_client/svs.sqlite does not exist (do not delete the entire ~/.securedrop_client directory)
  • the sd-devices VM is not running (shut down manually if necessary)
  • a supported printer is available, but not attached.
  • all VMs are up-to-date
  • test instance contains several sources, including some with files & some with HTML characters in messages

Login

  • when SecureDrop desktop icon is double-clicked, preflight updater is displayed
  • After preflight updater runs, when user clicks Continue, login dialog is displayed
  • In login dialog:
    • show/hide password functionality works
    • incorrect password cannot log in
    • 2FA token reuse cannot log in after password failure
    • invalid 2FA token cannot log in
    • valid credentials and 2FA can log in

Sources

  • after valid login:
    • the login dialog closes
    • source data is downloaded and source list is populated
    • user is prompted for GPG key access
    • submissions and replies are decrypted
    • the source list is displayed but no sources are selected by default
    • the conversation view is not populated
  • when a source is selected in source list:
    • conversation view is populated with source conversation
    • a source message containing HTML is displayed as unformatted text
    • source submissions have an active Download button
    • source submission compressed file size is displayed accurately
  • when the upper right 3-dot button is clicked:
    • a menu is displayed with a delete source account option
    • when delete source account is selected:
      • the source is deleted from the source list and the conversation view is blanked
      • the source is deleted from the server and not restored on next sync
      • source submissions and messages are removed from the client's data directory
  • when a source is starred in source list, and the client is closed and reopened in Online mode:
    • the source is still starred in the source list

Replies

  • when a source is selected in the source list:
    • the reply panel is available for use and there is no message asking the user to sign in
    • a reply can be added to the conversation
    • a pending reply can be added to the conversation (for development environments, you can use: wget https://gist.githubusercontent.com/creviera/7f19a7d10334359f40dbdbb2354cd13a/raw/a2ef94913a155aa4019b753cf916f844c9cffa3a/pending-reply && git apply pending-reply then send a reply; alternatively, disconnect the network or sd-whonix after sending a reply)
    • a failed reply can be added to the conversation (for development environments, you can use: wget https://gist.githubusercontent.com/creviera/5ba70d50c12b6a6df6f98ed40ad09645/raw/5caef3339ceab1fc997ccb6b9e337bc8828ef12f/failed-reply && git apply failed-reply then send a reply; alternatively, sign out after the previous step to confirm that the reply transitions to "failed" state)
    • a reply containing HTML is displayed as unformatted text
    • a reply with a single string of characters longer than 100 chars is displayed, but truncated (https://github.com/freedomofpress/securedrop-client/issues/815).
    • a reply with a line longer than 100 chars is displayed correctly
    • two replies added immediately after each other are ordered correctly

Submissions

For sample files in different formats, see this GDrive folder.

Preview
  • when Download is clicked on a submission:
    • the submission is downloaded and decrypted
    • the Download button is replaced with Print and Export options
    • the submission filename is displayed.
  • For a DOC submission:
    • when the submission filename is clicked, a disposable VM (dispVM) is started.
    • after the dispVM starts, the submission is displayed in LibreOffice
    • when LibreOffice is closed, the dispVM shuts down
  • For a PDF submission:
    • when the submission filename is clicked, a dispVM is started.
    • after the dispVM starts, the submission is displayed in evince
    • when evince is closed, the dispVM shuts down
  • For a JPEG submission:
    • when the submission filename is clicked, a dispVM is started.
    • after the dispVM starts, the submission is displayed in Image Viewer
    • when Image Viewer is closed, the dispVM shuts down
  • For an audio submission:
    • when the submission filename is clicked, a dispVM is started.
    • After the dispVM starts, the submission is played in Audacious
    • Sound is audible
    • when Audacious is closed, the dispVM shuts down
  • For a video submission:
    • when the submission filename is clicked, a dispVM is started.
    • After the dispVM starts, the submission is played in Totem
    • Sound is audible if applicable
    • when Totem is closed, the dispVM shuts down
  • For a compressed (archive) submission:
    • when the submission filename is clicked, a dispVM is started.
    • After the dispVM starts, the submission is opened in FileRoller
    • Individual files can be extracted and previewed
    • when FileRoller is closed, the dispVM shuts down
Batch submission actions
  • For a given source:
    • "Download all" is available
    • "Export all" is available
    • "Export all" shows a modal warning when not all files have been downloaded prior to export
      • Accepting the warning opens the export dialog (there is a known UI bug with export dialog size when the dialog is launched this way)
    • "Export conversation transcript" generates and exports a plaintext transcript that includes message text and file information
      • For downloaded files, the filename as submitted is included in the transcript
Export
  • When Export is first clicked on a submission:
    • the "Preparing to export..." message is displayed
    • the sd-devices VM is started
    • the user is prompted to insert an Export USB
    • On clicking Cancel, the prompt closes and the file is not exported
  • When Export is clicked on the submission again:
    • the "Preparing to export..." message is displayed
    • the user is prompted to insert an Export USB
    • When the user inserts an invalid Export USB, attaches it to the sd-devices VM and clicks Next:
      • a message is displayed indicating that the Export USB is invalid and the user is prompted to insert a valid device
  • When Export is clicked on the submission again:
    • the "Preparing to export..." message is displayed
    • the user is prompted to insert an Export USB
    • When the user inserts a valid Export USB, attaches it to the sd-devices VM, and clicks Next:
      • the user is prompted for the Export USB's password if the (LUKS) device is locked
      • the user is not prompted for the USB's password, if the (LUKS or VeraCrypt) device is unlocked
      • the user is not prompted for the USB's password, if the (LUKS or VeraCrypt) device is unlocked and mounted
    • When the user enters an invalid Export USB password and clicks Next:
      • a failure message is displayed and the user is prompted to enter the password again
    • When the user enters an valid Export USB password and clicks Next:
      • the file is saved to the Export USB and a success message is shown
    • When the user encounters error state(s) during export:
      • a user-facing message (rather than an EXPORT_ERROR_CODE style message) is shown
  • When the user detaches the Export USB and mounts it on another VM or computer:
    • the decrypted submission(s) is available in on the Export USB, in a directory sd-export-<timestamp>/export_data
Print
  • When the user clicks Print on a downloaded submission:
    • a "Preparing to print..." message is displayed
    • the sd-devices VM is started
    • the user is prompted to connect a supported printer
  • When the user connects a printer, attaches it to the sd-devices VM, and clicks Continue:
    • a "Printing..." message is displayed
    • the X Printer Panel dialog is displayed with the printer selected
  • When the user clicks Print in the X Printer Panel:
    • the submission is printed successfully.
  • A multi-page document can be printed successfully

Closing the client

  • When the user clicks the main window close button:
    • the client exits.

Scenario: Offline mode without existing data

Prerequisites:

  • server is available and contains source test data
  • test data includes at least one previously downloaded submission
  • test data includes at least one undownloaded submission
  • ~/.securedrop_client/data in sd-app is empty, and ~/.securedrop_client/svs.sqlite does not exist (do not delete the entire ~/.securedrop_client directory)
  • the sd-devices VM is not running (shut down manually if necessary)
  • a supported printer is available, but not attached.

Offline to Online

  • When SecureDrop desktop icon is double-clicked, preflight updater is displayed
  • After preflight updater runs, when user clicks Continue, login dialog is displayed
  • When user clicks Work Offline, login dialog closes and main window opens
  • after startup:
    • there is no sync attempt with the server
    • the source list is empty
  • When the user clicks the top-left user icon and chooses Sign in:
    • the login dialog is displayed over the main window
  • When the user enters valid login details and clicks Log in:
    • the login dialog closes
    • The user icon is updated to reflect the user's details
    • the client is synced with the server and the source list is updated
  • When the user selects a source with submissions from the source list:
    • the conversation view is populated with the source conversation
    • the reply panel is active
    • a reply can be sent to the source
    • a submission can be downloaded
    • a downloaded submission can be exported
  • When the user clicks the main window close button:
    • the client exits.

Scenario: Offline mode with existing data

Prerequisites:

  • server is available and contains source test data
  • test data includes at least one previously downloaded submission
  • test data includes at least one undownloaded submission
  • client data directory has been synced with server in a previous login
  • the sd-devices VM is not running (shut down manually if necessary)
  • a supported printer is available, but not attached.

Offline to Online

  • When SecureDrop desktop icon is double-clicked, preflight updater is displayed
  • After preflight updater runs, when user clicks Continue, login dialog is displayed
  • When user clicks Work Offline, login dialog closes and main window opens
  • after startup:
    • there is no sync attempt with the server
    • the source list is populated with contents of last server sync
  • When the user selects a source with submissions from the source list:
    • the conversation view is populated with the source conversation
    • the reply panel is inactive, with a "Sign in" message
    • a previously downloaded submission can be exported
    • a previously downloaded submission can be printed
    • When the user clicks Download on an undownloaded submission, a message is displayed instructing the user to sign in to perform the download
  • When the user clicks the top-left user icon and chooses Sign in:
    • the login dialog is displayed over the main window
  • When the user enters valid login details and clicks Log in:
    • the login dialog closes
    • The user icon is updated to reflect the user's details
    • source data is synced with the server
  • When the user selects a source with submissions from the source list:
    • the conversation view is populated with the source conversation
    • the reply panel is active
    • When the user replies to a source, the reply is added to the source conversation
    • When the user clicks Download on an undownloaded submission, the submission is downloaded and decrypted
    • When the user clicks Export on a submission, the export process can be completed
    • When the user clicks Print on a submission, the print process can be completed
  • When the user clicks the main window close button:
    • the client exits.

Scenario: Client and Journalist Interface both in use

Note: this scenario requires access to the Journalist Interface (JI) via Tor Browser. If the scenario is being tested on Qubes, the JI address can be found in sd-whonix in /usr/local/etc/torrc.d/50_user.conf. See https://github.com/freedomofpress/securedrop-workstation/wiki/Developer-Tips#how-to-connect-to-the-journalist-interface-in-qubes for instructions on how to connect to the JI in a VM.

Prerequisites:

  • server is available and contains source test data
  • client data directory is empty

Login

  • when SecureDrop desktop icon is double-clicked, preflight updater is displayed
  • After preflight updater runs, when user clicks Continue, login dialog is displayed
  • after valid login to client:
    • the login dialog closes
    • source data is downloaded and source list is populated
    • user is prompted for GPG key access
    • submissions and replies are decrypted
    • the source list is displayed but no sources are selected by default
    • the conversation view is not populated
  • when the JI address is visited in Tor Browser:
    • JI login page is displayed
  • after valid login to JI using same account as for client:
    • sources page is displayed, containing the same sources as the client (order may differ)

Sources, replies, submissions

  • when a source is starred in the client:
    • the source is also starred in the JI after a page reload
  • when a starred source is unstarred in the JI:
    • the source is also unstarred in the client after next sync.
  • when a reply is sent to a source via the client:
  • the reply is visible in the JI and can be viewed by the source in the Source Interface
  • when a reply is sent to a source via the JI:
    • the reply is visible in the source conversation view after next sync
  • when the journalist account used to reply is deleted by an admin in the JI:
    • the next sync is successful
    • the reply is visible in the conversation view
    • the journalist's details are deleted from the client database
  • when a reply is deleted by a source:
  • when an individual file submission is deleted in the JI:
    • the submission is no longer listed in the conversation view
    • the submission files are deleted from the client data directory
  • when an individual message is deleted in the JI:
    • the message is no longer listed in the conversation view
    • the messages are deleted from the client database
  • when a source is deleted in the JI:
    • the source is no longer listed in the client after next sync
    • files associated with the source are no longer present in the client data directory
  • when a source is deleted in the client:
    • the source is no longer listed in the JI after a page reload

Scenario: Large dataset

Prerequisites:

  • server is available and contains large source test dataset (256 sources, submission sizes ranging from 1-500MB)
  • client data directory is empty

Sources

  • after valid login:
    • the login dialog closes
    • all source data is downloaded and source list is populated
    • user can scroll to bottom of source list
    • user is prompted for GPG key access
    • submissions and replies are decrypted
    • the source list is displayed but no sources are selected by default
    • the conversation view is not populated
  • when a source is selected in source list:
    • conversation view is populated with source conversation
    • a source message containing HTML is displayed as unformatted text
    • source submissions have an active Download button
    • source submission compressed file size is displayed accurately
  • when the upper right 3-dot button is clicked:
    • a menu is displayed with a delete source account option
    • when delete source account is selected:
      • the source is deleted from the source list and the converation view is blanked
      • the source is deleted from the server and not restored on next sync
      • source submissions and messages are removed from the client's data directory
  • when a source is starred in source list, and the client is closed and reopened in Online mode:
    • the source is still starred in the source list

Replies

  • when a source is selected in the source list:
    • the reply panel is available for use and there is no message asking the user to sign in
    • a reply can be added to the conversations
    • a reply containing HTML is displayed as unformatted text
    • two replies added immediately after each other are ordered correctly

Submissions

Preview
  • when Download is clicked on a submission:
    • the submission is downloaded and decrypted
    • the Download button is replaced with Print and Export options
    • the submission filename is displayed.
  • For a DOC submission:
    • when the submission filename is clicked, a disposable VM (dispVM) is started.
    • after the dispVM starts, the submission is displayed in LibreOffice
    • when LibreOffice is closed, the dispVM shuts down
  • For a PDF submission:
    • when the submission filename is clicked, a dispVM is started.
    • after the dispVM starts, the submission is displayed in evince
    • when evince is closed, the dispVM shuts down
  • For a JPEG submission:
    • when the submission filename is clicked, a dispVM is started.
    • after the dispVM starts, the submission is displayed in Image Viewer
    • when evince is closed, the dispVM shuts down

Release-specific test plans

Some of the tests below should be incorporated into main test plan after the release, while others will not need to be re-tested with each release.

SecureDrop Workstation 0.3.0

Moved to https://github.com/freedomofpress/securedrop-workstation/issues/548

SecureDrop Client 0.2.0

Moved to https://github.com/freedomofpress/securedrop-client/pull/1083

Clone this wiki locally