Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change usage reporting defaults backwards-incompatibly to send less data #6051

Closed
glasser opened this issue Jan 26, 2022 · 5 comments
Closed

Change usage reporting defaults backwards-incompatibly to send less data #6051

glasser opened this issue Jan 26, 2022 · 5 comments
Assignees
@glasser
Milestone
Release 4.0

Comments

@glasser
Copy link
Member

glasser commented Jan 26, 2022

In order to make usage reporting have better performance and less potential PII leakage by default, we should change the default usage reporting to send even less information by default. We previously changed usage reporting to make variable and HTTP header reporting opt-in. We should continue down this path to make sending error bodies in traces opt-in (though we should still probably track the number of errors by default). Furthermore, now that the Unified Reporting and referenced field usage projects are done, we can change the defaults to make traces be an opt-in feature (which gives you access to traces on the Operations page, field execution counts on the Fields page, and field latencies in Explorer and VSCode timing hints) that isn’t on by default. This would mean that by default we would only send summarized statistics of operations, not any data about individual requests, and the only user-provided strings would be operation signatures (with literals removed as always).
@glasser glasser added this to the Release 4.0 milestone Jan 26, 2022
@glasser glasser self-assigned this Jan 26, 2022
@glasser glasser added rebase and removed rebase labels May 12, 2022
@glasser glasser mentioned this issue Jul 21, 2022
Closed
@glasser
Copy link
Member Author

glasser commented Jul 21, 2022

See also #6708.

glasser added a commit that referenced this issue Aug 10, 2022
@glasser
usage reporting/inline trace plugins: mask errors, change option name
455b339 
We replace the `rewriteError` option with `sendErrorsInTraces` (usage
reporting) and `includeErrors` (inline trace), which takes `{unmodified:
true}`, `{masked:true}`, and `{transform}` variants. This is similar to
`sendVariableValues` and `sendHeaders`.

Like those two options, this now defaults to a more redacted version:
`{masked: true}`. This will reduce unintentional reporting of PII.

Part of #6051.

Paired with @bonnici.
glasser added a commit that referenced this issue Aug 10, 2022
@glasser
usage reporting/inline trace plugins: mask errors, change option name
6268bab 
We replace the `rewriteError` option with `sendErrorsInTraces` (usage
reporting) and `includeErrors` (inline trace), which takes `{unmodified:
true}`, `{masked:true}`, and `{transform}` variants. This is similar to
`sendVariableValues` and `sendHeaders`.

Like those two options, this now defaults to a more redacted version:
`{masked: true}`. This will reduce unintentional reporting of PII.

Part of #6051.

Paired with @bonnici.
@glasser glasser mentioned this issue Aug 10, 2022
Merged
glasser added a commit that referenced this issue Aug 10, 2022
@glasser
usage reporting/inline trace plugins: mask errors, change option name (…
7445d33 
…#6794)

We replace the `rewriteError` option with `sendErrorsInTraces` (usage
reporting) and `includeErrors` (inline trace), which takes `{unmodified:
true}`, `{masked:true}`, and `{transform}` variants. This is similar to
`sendVariableValues` and `sendHeaders`.

Like those two options, this now defaults to a more redacted version:
`{masked: true}`. This will reduce unintentional reporting of PII.

Part of #6051.

Paired with @bonnici.
@glasser
Copy link
Member Author

glasser commented Aug 16, 2022

Hmm. If errors are masked, should we override the logic in defaultSendOperationAsTrace that treats traces with errors specially?

@glasser
Copy link
Member Author

glasser commented Aug 26, 2022

Hmm. If errors are masked, should we override the logic in defaultSendOperationAsTrace that treats traces with errors specially?

We can make this change at any time without compat worries because that heuristic is deliberately unspecified.

In the interest of moving this project towards finished, let's leave the defaults for "send any traces?" and "instrument fields?" at turned on. The onboarding experience is better that way (ie, no changes needed to Studio UI) and if it's a performance problem you can tweak the settings (or move to Router if you're on Gateway!).

@glasser glasser closed this as completed Aug 26, 2022
@glasser glasser reopened this Aug 26, 2022
@glasser
Copy link
Member Author

glasser commented Aug 26, 2022

That said:
Right now there isn’t an easy clearly supported way to say “I just don’t want to send traces at all”. There’s experimental_sendOperationAsTrace which is, well, experimental (and completely replaces the main trace vs stats heuristic). And there’s fieldLevelInstrumentation: false which sort of does that but it will still send traces with (masked) errors for parse and validation failures. So while I’m not going to change the default behavior, I am going to add a little sendTraces: false setting that you can do to turn off sending anything as a trace (but it doesn’t imply fieldLevelInstrumentation: false! you can still get field execution stats from this!)

glasser added a commit that referenced this issue Aug 26, 2022
@glasser
usage reporting: add sendTraces, rename sendErrorsInTraces, etc
6cd6f7c 
- We've decided to leave the usage reporting defaults in AS4 as "field
  level instrumentation on for all operations, send some operations as
  traces". But we'd like to at least provide a non-experimental
  mechanism for entirely disabling sending traces to Studio. So there's
  a new ApolloServerPluginUsageReporting: `sendTraces: false`.
- Rename the (new in AS4) usage reporting option `sendErrorsInTraces` to
  `sendErrors`, because it does also affect error statistics in stats
  reports if your `transform` function returns `null`.
- Remove Apollo-internal `internal_includeTracesContributingToStats`
  option. This enabled some internal consistency monitoring which we are
  no longer paying attention to.
- If you enable `debugPrintReports`, send the reports as `info` rather
  than `warn`, which primarily lets us delete some large comments (and
  makes sense because these debug messages are not warnings).

Fixes #6051. Fixes #6078.
@glasser glasser mentioned this issue Aug 26, 2022
Merged
glasser added a commit that referenced this issue Aug 26, 2022
@glasser
usage reporting: add sendTraces, rename sendErrorsInTraces, etc
d155b36 
- We've decided to leave the usage reporting defaults in AS4 as "field
  level instrumentation on for all operations, send some operations as
  traces". But we'd like to at least provide a non-experimental
  mechanism for entirely disabling sending traces to Studio. So there's
  a new ApolloServerPluginUsageReporting: `sendTraces: false`.
- Rename the (new in AS4) usage reporting option `sendErrorsInTraces` to
  `sendErrors`, because it does also affect error statistics in stats
  reports if your `transform` function returns `null`.
- Remove Apollo-internal `internal_includeTracesContributingToStats`
  option. This enabled some internal consistency monitoring which we are
  no longer paying attention to.
- If you enable `debugPrintReports`, send the reports as `info` rather
  than `warn`, which primarily lets us delete some large comments (and
  makes sense because these debug messages are not warnings).

Fixes #6051. Fixes #6078.
glasser added a commit that referenced this issue Aug 26, 2022
@glasser
usage reporting: add sendTraces, rename sendErrorsInTraces, etc
596b4f6 
- We've decided to leave the usage reporting defaults in AS4 as "field
  level instrumentation on for all operations, send some operations as
  traces". But we'd like to at least provide a non-experimental
  mechanism for entirely disabling sending traces to Studio. So there's
  a new ApolloServerPluginUsageReporting: `sendTraces: false`.
- Rename the (new in AS4) usage reporting option `sendErrorsInTraces` to
  `sendErrors`, because it does also affect error statistics in stats
  reports if your `transform` function returns `null`.
- Remove Apollo-internal `internal_includeTracesContributingToStats`
  option. This enabled some internal consistency monitoring which we are
  no longer paying attention to.
- If you enable `debugPrintReports`, send the reports as `info` rather
  than `warn`, which primarily lets us delete some large comments (and
  makes sense because these debug messages are not warnings).

Fixes #6051. Fixes #6078.
glasser added a commit that referenced this issue Aug 26, 2022
@glasser
usage reporting: add sendTraces, rename sendErrorsInTraces, etc (#6855)
3e4ab3f 
- We've decided to leave the usage reporting defaults in AS4 as "field
  level instrumentation on for all operations, send some operations as
  traces". But we'd like to at least provide a non-experimental
  mechanism for entirely disabling sending traces to Studio. So there's
  a new ApolloServerPluginUsageReporting: `sendTraces: false`.
- Rename the (new in AS4) usage reporting option `sendErrorsInTraces` to
  `sendErrors`, because it does also affect error statistics in stats
  reports if your `transform` function returns `null`.
- Remove Apollo-internal `internal_includeTracesContributingToStats`
  option. This enabled some internal consistency monitoring which we are
  no longer paying attention to.
- If you enable `debugPrintReports`, send the reports as `info` rather
  than `warn`, which primarily lets us delete some large comments (and
  makes sense because these debug messages are not warnings).

Fixes #6051. Fixes #6078.
@glasser
Copy link
Member Author

glasser commented Aug 26, 2022

Finished up in #6855.

@glasser glasser closed this as completed Aug 26, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@glasser glasser

Labels
None yet
Projects
None yet
Milestone
Release 4.0
Development

No branches or pull requests
1 participant
@glasser