Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add guide for app cloud testing #25900

Merged
merged 2 commits into from
Feb 22, 2023
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions guides/app-cloud-testing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# App <--> Cloud Testing

Testing communication between the Cypress App and Cypress Cloud can be complex depending on how much data is going back and forth. For example, we recently completed the Debug page feature which involved writing some e2e tests to verify that the page was functioning correctly with the expected data from Cypress Cloud. In this document, we'll share some of our findings from the process of writing these tests for the Debug page.

## Working with GraphQL requests

We use GraphQL to interface with Cypress Cloud, so there are some things to know when writing e2e tests between the App and Cloud.

### GraphQL requests over fetch

By default, GraphQL queries use web sockets to fetch data. This doesn't let us intercept these messages and return a mock response in the ideal way (using cy.intercept) to get our app into the desired state for the test. We can set the GraphQL client to use fetch rather than WS by adding this hook at the top of our spec file.

You can see an example of this in [our spec for the Debug page](/packages/app/cypress/e2e/debug.cy.ts#L4).

```js
Cypress.on('window:before:load', (win) => {
win.__CYPRESS_GQL_NO_SOCKET__ = 'true'
})
```

Note that this does not affect [GraphQL subscriptions](#intercepting-graphql-subscriptions).

Now that the GraphQL requests to Cypress Cloud are happening over fetch, we can use `cy.intercept` to intercept the requests and do whatever we need to do (in most cases, return a JSON fixture as a response to the request).

### Creating JSON fixtures from GraphQL requests

Another benefit of making our GraphQL requests over fetch is that we can easily see the real responses from Cypress Cloud in the Network tab of our browser developer tools. This is especially useful when we have large responses coming back from Cypress Cloud that we want to mock in our tests.

We can start our fixture by copying the real response from the dev tools response into our fixture and then modifying individual fields to tweak our app state. Then we can intercept the GraphQL request and return our fixture.

You can see an example of this in [our spec for the Debug page](/packages/app/cypress/e2e/debug.cy.ts#L35).

```js
cy.intercept('query-Debug', {
fixture: 'debug-Passing/gql-Debug.json',
})
```

### Intercepting GraphQL subscriptions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a great understanding of the graphql intercepts but looking through the code I don't think that cy. remoteGraphQLIntercept has anything to do with subscriptions but rather the ability to intercept queries made to the cloud that originate from the server.

For this example, a subscription wired to pollForRuns triggers a cloud query from RelevantRunsDataSource.ts. If we want to intercept this query, we need to use remoteGraphQLIntercept. The subscription is just an implementation detail. If it was hooked up to a mutation rather than a subscription, we would be ok just using cy.intercept.

I think the distinction is important based on how much coverage we want. In true e2e fashion, if we want to test that our code inside of data-context is wired up properly we would leverage cy.remoteGraphQLIntercept as this would allow the data-context methods to run and the only thing stubbed would be the cloud call.

The description of this suggests we could use cy.remoteGraphQLIntercept to intercept the specs change subscription but I don't think that is possible (rather, we could reach into data-context via cy.withCtx and trigger a sub manually).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah, I think I misunderstood what was going on when I reviewed Lachlan's PR for these tests. What's happening is that this query is coming from RelevantRunsDataSource.ts, so cy.intercept won't work and we need to use remoteGraphQLIntercept. The fact that it's a subscription doesn't have anything to do with it.

I'm confused about this part

If it was hooked up to a mutation rather than a subscription, we would be ok just using cy.intercept.

So if the mutation originated from the server, we'd still be able to intercept it using cy.intercept?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry should have been more clear. I meant that if there was a frontend mutation that triggered a cloud query, we could intercept the mutation and stub the result essentially bypassing the cloud query (as the methods defined in data-context for initiating the cloud query would never run). Since this example was wired up to subscriptions, we can't do that.

For the most accurate coverage I'd say we shouldn't rely on cy.intercept as that skips the graphql resolvers that are hooked up to our data-context methods, but getting the initial state of data-context setup properly could be messy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay. So is your thought that we should either use cy.remoteGraphQLIntercept or cy.withCtx instead of trying to intercept the requests over fetch with cy.intercept?

Copy link
Contributor

@ZachJW34 ZachJW34 Feb 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say yes but I don't want to pivot from what we currently have. I think we can make a note mentioning the cons of using cy.intercept (we lose coverage of our graphql resolvers and data-context calls) and fix up this section to state the correct definition/usage of cy.remoteGraphQLIntercept

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you can cy.intercept a subscription since it goes over websockets - this is why I had to remoteIntercept, though what @ZachJW34 says is true - the main difference is remoteIntercept is between Server<->Cloud, but intercept between App and server.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can use cy.intercept for subscriptions, that sure would be neat.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I made some updates


Subscriptions always use web sockets, so in order to intercept these we need to do something different. For subscriptions, we need to use `cy.remoteGraphQLIntercept` to check if the operation is related to our subscription and if it is, return our fixture instead.

You can see an example of this in [our spec for the Debug page](/packages/app/cypress/e2e/debug.cy.ts#L23).

```js
import RelevantRunsDataSource_RunsByCommitShas from '../fixtures/gql-RelevantRunsDataSource_RunsByCommitShas.json'

beforeEach(() => {
cy.remoteGraphQLIntercept((obj, _testState, options) => {
if (obj.operationName === 'RelevantRunsDataSource_RunsByCommitShas') {
obj.result.data = options.RelevantRunsDataSource_RunsByCommitShas.data
}

return obj.result
}, { RelevantRunsDataSource_RunsByCommitShas })
})
```