Replace Hangfire with a simpler queue service

[mscottford]

We're currently using Hangfire for processing IApplicationEvent and IApplicationActivity instances. This is a great library that provides a lot of functionality, and it was decently easy for us to get started with it. It does come with some downsides.

• Serialization overhead: all work that's stored in the job queue has to be serialized as it is being stored in the queue and deserialized as it is read out of the queue. This is great for applications that need to persist their work in a shared storage mechanism so that it can be read by another process. However, in our case, all of our work happens in the same process, so it would make more sense to keep all of our work in memory instead. If we could do that, we'd improve performance by not having to serialize and deserialize everything going into and coming out of the work queue. Hangfire also requires us to depend on the Newtonsoft JSON library which is not as performant as System.Text.Json.
• Complicated setup: Because of the way that Hangfire is designed, we had to extract (read: copy and paste) a lot of ASP.NET specific setup code and port it to work without ASP.NET. This makes the Hangfire a rather brittle dependency.
• Synchronous work: Hangfire does not provide a way for us to add asynchronous methods to the work queue. This means that we're not able to take advantage of async and await to better mange CPU resources. As a result, a lot of our CPU usage percentage is caused by us just waiting on IO operations to complete. It would be much more performant to be able to leverage asnyc method calls for those cases.
• Work stored as expressions: Because the work that goes in the queue has to be represented by normal expressions, we've had to implement some strange work arounds that will break in unobvious ways if someone refactors the wrong method.

A solution that's worth investigating is building our own queue system using an unbounded System.Threading.Channel. to store the work that needs to be completed. The work can be stored simple Task<T> objects. Microsoft has a really decent guide that we can use as a starting point for this effort. We'd then be able to convert our IApplicationActivity and IApplicationEvent classes to provide async capable methods so that we can obtain further performance improvements.