Exception Notifications in Slack

[saneshark]

Currently dev and staging environments suppress the backtrace from exceptions raised because that is the preference for production. But for debugging, it is incredibly helpful to have this backtrace. In lieu of a mechanism for querying logs in a realtime manner, this is a good compromise for now.

This PR can be merged irrespective of https://github.com/department-of-veterans-affairs/devops/pull/685 but will not start to track the exceptions in slack until those ENV variables are present.

[earthboundkid]

In past jobs, we have used Sentry for this. Slack seems like a less useful choice because Sentry is designed for the job and can customize how many emails get sent, store related stack traces together, etc. What services for error reporting have we looked at? I believe CloudWatch has been mentioned.

[markolson]

I'd really rather use a dedicated tool for exception handling, and then that tool can use whatever Slack integration to inform us of new exceptions.

[saneshark]

Sentry is on its way but could be 1-2 weeks out. This is an interim solution because QA is right around the corner, in many instances these 500 errors already occur an no one has time to speculate on what raised them.

[saneshark]

@markolson what would be an example of a dedicated tool for exception handling? NewRelic?

[saneshark]

See the thread on issue 103 for example

[ayaleloehr]

Sorry @saneshark but I'm going to close this. My gut is saying that automatically posting log messages from GovCloud to Slack has too high a probability to get us in trouble to be worth using as a stopgap measure until Sentry is available. I have access to AWS logs and if I can get timestamps for any exceptions you're looking for, I can pull the logs and pass them along.

@jkassemi is prioritizing Sentry as fast as the VA will let him go (AKA he is waiting on DNS).