Add a queue for HCAD model cold start.

[kaituo]

Note: since there are a lot of dependencies, I only list the main class and test code to save reviewers' time. The build will fail due to missing dependencies. I will use that PR just for review. will not merge it. Will have a big one in the end and merge once after all review PRs get approved. Now the code is missing unit tests. Posting PRs now to meet the cutoff date (June 1). Will add unit tests, run performance tests, and fix bugs before the official release.

Description

This PR adds a queue for HCAD model training (a.k.a. cold start). As model training is a pretty expensive operation, we pull cold start requests from the queue in a serial fashion. Each detector has an equal chance of being pulled. The equal probability is achieved by putting model training requests for different detectors into different segments and pulling requests from segments in a round-robin fashion.

To accommodate the model training queue, we need to add a listener as an input to EntityColdStarter's trainModel method, the utility class for model training. The listener needs to be invoked whenever EntityColdStarter finishes training or encounters exceptions. The listener helps notify the cold start queue to pull another request (if any) to execute.

Some of the rate-limiting logic in EntityColdStarter is removed as we have a cold start queue now.

We have also changed to use DoorKeeper to limit the cold start an entity can trigger in a certain period. DoorKeeper is a bloom filter that is used elsewhere in the package. Reusing the logic reduces the amount of code we have to maintain. The DoorKeeper needs to be constantly maintained. Thus, I let EntityColdStarter implement MaintenanceState and make CronTransportAction clean DoorKeeper regularly.

Testing done:

1. Manual tests using 10 HCAD detectors and 12,000 entities in a 3 node cluster.

Check List

• [ Y ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.