Add StatsD tracking

[mxstbr]

Status

• WIP
• Ready for review
• Needs testing

Deploy after merge (delete what needn't be deployed)

• api
• hyperion (frontend)

Adds StatsD tracking to our HTTP servers (api and hyperion).

Todo

• Set up a middleware to track request times
• Handle GraphQL operation tracking
• Submit metrics to DataDog in production
• Log metrics in development

I'm currently waiting to get the DataDog API key to post this data to DataDog!

[mxstbr]

This is what the logs look like for hyperion (the server-side renderer) and the API:

hyperion.http.get.spectrum.general.response_time:3012|ms
hyperion.http.get.spectrum.general.status_code.200:1|c

api.http.post.api.getCommunityThreads.response_time:35|ms
api.http.post.api.getCommunityThreads.status_code.200:1|c

Note how it adds the GraphQL operation name to the API logs, which should make it easier to debug expensive GraphQL queries and optimize them, and the request pathname to all requests.

[mxstbr]

Latest couple commits switch to a fork of express-statsd I just made (https://github.com/withspectrum/express-hot-shots) to add support for tags in the metrics, which allows us to tag the GraphQL operation name instead of putting it into the metric key as recommended by the observability team! 💯

That means our metrics now look like this, note that the server name and operation name are moved to the tags (after the #):

http.get.spectrum.general.id-123.response_time:3012|ms|#server:hyperion
http.get.spectrum.general.id-123.status_code.200:1|c|#server:hyperion
db.query_queue.size:2|g|#server:hyperion

http.post.api.response_time:35|ms|#server:api,graphqlOperationName:getCurrentUser
http.post.api.status_code.200:1|c|#server:api,graphqlOperationName:getCurrentUser
db.connection_pool.size:58|g|#server:api

[mxstbr]

This is not quite ready yet, I was recommended to change towards tracking response times and status codes per view rather than per pathname (i.e. http.get.community, http.get.thread, http.get.dashboard instead of http.get.figma, http.get.figma.general.asdf-123). Going to dig into that, will take a bit more work since I have to reverse-engineer our React routing setup somehow.

[mxstbr]

Latest commit switches to https://github.com/dbader/node-datadog-metrics, which avoids the need to run the DataDog agent and simply pings the DataDog API directly! This works and is live in production for testing

[brianlovin]

@mxstbr looks like we just need to clean up flow and merge alpha, then let's merge this!

[mxstbr]

Have already learned a bunch of stuff from having this run in production, will rework a little bit to make it more useful for all the workers and then ship!

[mxstbr]

The latest commits rework the tracking a bit:

• Remove paths from the metrics: this is an anti-pattern according to the #observability folks, as it's a) not very useful (DataDog doesn't let you aggregate sub-metrics) and b) blows up storage size, making it very expensive. Rather than tracking e.g. http.get.auth.twitter.response_time we now simply track http.get.response_time across all our servers
• Track db query response size and time: we're now tracking how long database queries take and how much data they return, allowing us to notice abnormalities across all our servers much quicker
• Stop tracking connection pool size, as it's constant now
• Adds nicer logging in dev

I think this is probably good to ship, going to wait for tests to pass and do a test deploy on alpha first though!

[mxstbr]

This has been up on api.alpha for a bit and looks fine, wanna give it a review and then deploy to all the servers today @brianlovin? That'd be great

[brianlovin]

This is awesome @mxstbr - are you going to set up some sensible default dashboards in datadog that we can start using going forward? I imagine you have a more clear picture of how you'd like this reporting to be visualized and prioritized.

Talked offline: Max to create dashboards for us tomorrow :)