Plugin initialization and aggregation

[j-lanson]

Note: This PR relies on #278 which must be merged first.

This PR makes small adjustments to and expands upon #278 . The PluginExecutor now has a start_plugins() async function which allows a set of plugins to be started concurrently. For a simple implementation, this required removing mutable state from the PluginExecutor, hence the elimination of the est_ports field.

The HcPluginCore struct was added, which is intended to manage the protocol state of connections to each plugin and handle requesting info from a different plugin on behalf of another. The HcPluginCore::new() function takes an executor, and a set of (Plugin, Configuration) pairs. This function starts all the plugins and also initializes them over gRPC by getting the schemas, setting the config, and getting the default policy expression. In order to hold important state and distinguish between an initialized and uninitialized PluginContext, the PluginTransport struct was created. It contains the context as well as handles for sending/receiving Query messages with the plugin on the other side. The goal is for the chunking portion of the protocol to be hidden within this struct

[alilleybrinker]

We may in the future want to consider Tokio's JoinSet instead of join_all here. The reasons are subtle and will require careful consideration. First, I recommend reading the following:

• https://without.boats/blog/futures-unordered/
• https://without.boats/blog/let-futures-be-futures/

We can go forward with the current design for now, but I want us to be thinking about this.

[j-lanson]

We may in the future want to consider Tokio's JoinSet instead of join_all here. The reasons are subtle and will require careful consideration. First, I recommend reading the following:

• https://without.boats/blog/futures-unordered/
• https://without.boats/blog/let-futures-be-futures/

We can go forward with the current design for now, but I want us to be thinking about this.

I had looked at tokio::JoinSet without knowing the subtlety between the two, and I didn't like that it would force the PluginExecutor to be 'static when its behavior is anticipated to be controlled by the Hipcheck config file. The alternative is a global OnceLock type situation, but it's not one I relish.

[alilleybrinker]

We may in the future want to consider Tokio's JoinSet instead of join_all here. The reasons are subtle and will require careful consideration. First, I recommend reading the following:

• https://without.boats/blog/futures-unordered/
• https://without.boats/blog/let-futures-be-futures/

We can go forward with the current design for now, but I want us to be thinking about this.

I had looked at tokio::JoinSet without knowing the subtlety between the two, and I didn't like that it would force the PluginExecutor to be 'static when its behavior is anticipated to be controlled by the Hipcheck config file. The alternative is a global OnceLock type situation, but it's not one I relish.

Oh that's not what the 'static bound means. Any type which doesn't hold references is 'static, so what this means in practice is that if tasks want to hold pointers to state from elsewhere that access needs to be mediated by Arc and some sort of exclusion like a Mutex.

[j-lanson]

We may in the future want to consider Tokio's JoinSet instead of join_all here. The reasons are subtle and will require careful consideration. First, I recommend reading the following:

• https://without.boats/blog/futures-unordered/
• https://without.boats/blog/let-futures-be-futures/

We can go forward with the current design for now, but I want us to be thinking about this.

I had looked at tokio::JoinSet without knowing the subtlety between the two, and I didn't like that it would force the PluginExecutor to be 'static when its behavior is anticipated to be controlled by the Hipcheck config file. The alternative is a global OnceLock type situation, but it's not one I relish.

Oh that's not what the 'static bound means. Any type which doesn't hold references is 'static, so what this means in practice is that if tasks want to hold pointers to state from elsewhere that access needs to be mediated by Arc and some sort of exclusion like a Mutex.

Alternatively, the PluginExecutor does not currently contain mutable state, and its member fields are all simple primitive types (minus one Range object). So cloning the PluginExecutor and moving the clone into the Future could be another strategy. That seems like it would be more performant than each Future having to wait its turn just to get access to different config fields.

[alilleybrinker]

Ah, since it's configuration and/or policy data that is never modified, a OnceLock or LazyLock wrapped in an Arc would be good. Reducing the duplication seems more important than removing a small amount of pointer indirection and increment / decrement for sharing (which only happens at the beginning). OnceLock / LazyLock encode the "initialize once and then never modify again" pattern and mean that each task gets to basically act like it has a normal reference without infecting the code with lifetimes that violates the JoinSet rules.

[alilleybrinker]

LGTM! Just needs a rebase to squash to one commit. Ping me after that, I'll re-review, and we can merge!