Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error monitoring #651

Merged
merged 4 commits into from
Jul 23, 2021
Merged

Error monitoring #651

merged 4 commits into from
Jul 23, 2021

Conversation

tim-schilling
Copy link
Collaborator

@tim-schilling tim-schilling commented May 27, 2021

Monitor errors via got_request_exception signal.

The middleware process_exception would only allow us to track
exceptions thrown in views, that weren't picked up by an
earlier middleware that returned a result. This solution allows
us to track exceptions that occur anywhere in the application.

The downside to this is that it's possible if another receiver
of got_request_exception throws a new exception, our handler
may get the wrong exception with sys.exc_info().

We could limit the scope of that issue by injecting a middleware
that's as far down the list as possible to wrap the view
and catch exceptions there. Then rely on the signal handler
for catching exceptions in the middleware.

Create ErrorServiceThread to buffer error messages.

TODO:

  • Django instrumentation unit tests
  • ErrorMonitor core unit tests
  • Documentation of new settings and error monitoring feature.
  • Batch errors together and send on separate thread from the requests thread.
  • API for manually tracking errors

@tim-schilling tim-schilling mentioned this pull request May 27, 2021
5 tasks
@tim-schilling tim-schilling force-pushed the error-monitoring branch 4 times, most recently from 887c136 to 325dbdd Compare July 7, 2021 19:27
@tim-schilling tim-schilling marked this pull request as ready for review July 7, 2021 19:27
@tim-schilling tim-schilling changed the title WIP - Error monitoring Error monitoring Jul 7, 2021
@tim-schilling tim-schilling requested a review from dlanderson July 7, 2021 19:29
@tim-schilling tim-schilling force-pushed the error-monitoring branch 2 times, most recently from 1d3db33 to e0e4b6b Compare July 16, 2021 14:12
@dlanderson
Copy link
Contributor

Where does ErrorServiceThread get started? Does it start even if errors_enabled config is false?

@tim-schilling
Copy link
Collaborator Author

Where does ErrorServiceThread get started? Does it start even if errors_enabled config is false?

ErrorServiceThread.ensure_started is called at the end of ErrorServiceThread.send. This keeps the thread from being created until it's needed.

@dlanderson
Copy link
Contributor

@tim-schilling If we've tested and all looks good and you're ready to release let's go ahead 👍

@tim-schilling
Copy link
Collaborator Author

I'm going to push the existing changes out in a version, then merge this as another version.

@tim-schilling tim-schilling force-pushed the error-monitoring branch 2 times, most recently from 1b6fe64 to 9a4b91d Compare July 23, 2021 13:45
The middleware process_exception would only allow us to track
exceptions thrown in views, that weren't picked up by an
earlier middleware that returned a result. This solution allows
us to track exceptions that occur anywhere in the application.

The downside to this is that it's possible if another receiver
of got_request_exception throws a new exception, our handler
may get the wrong exception with sys.exc_info().

We could limit the scope of that issue by injecting a middleware
that's as far down the list as possible to wrap the view
and catch exceptions there. Then rely on the signal handler
for catching exceptions in the middleware.

Create ErrorServiceThread to buffer error messages.

Implement buffering logic, write integration tests for error_service.

Exclude specific libraries' logs from caplog.

Some of our checks are positional based, but have the chance of triggering
a libraries own debug statements. For cases we know we can ignore,
we can add them to the filter to make the tests more robust.

Expose a configurable API and ErrorMonitor interface.

Support Celery for the error monitoring solution.

Custom params belong on the context.

Celery's custom params go into a celery key of custom params.
Custom controller overrides the request_component's controller.
Deprecates backtrace.capture in favor of backtrace.capture_backtrace.
@tim-schilling tim-schilling merged commit 4b8ef04 into master Jul 23, 2021
@tim-schilling tim-schilling deleted the error-monitoring branch July 23, 2021 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants