perf: Adds a caching to shard management counts to alleviate shard scanning O(n)

[astubbs]

Description...

Checklist

• Documentation (if applicable)
• Changelog
• Final speed comparison to origin

[what-the-diff]

• Added a new run configuration for performance tests
• Updated the pom file to include ProgressBar class in bytecode enhancement process
• Removed log statement from poll method of ParallelEoSStreamProcessor class as it was causing too much logging and slowing down test execution time by ~10% (Encoding back pressure system so that offset payloads are prevented from being too large #47)
• Fixed bug where interrupting control thread would cause an exception if work mailbox is not being polled currently, which could happen when there are no records available on broker or all partitions have been revoked due to rebalance event (https://github.com/confluentinc/parallel-consumer#bugfix--interrupting-controlthread---exception)
• & 5b: Changed offset map encoding logic so that partition state can be set into blocked mode once payload size exceeds pressure threshold value but still lower than max allowed metadata size limit; this will prevent further messages from being processed until record success reduces encoded payload size below pressure threshold value again - https://github.com//confluentinc//parallel-consumer/#feature--offsetmapencodingbackpressurethresholds
• The function workIsWaitingToBeProcessed() was renamed to isWorkWaitingToBeProcessed().
• A new field totalShardEntriesNotInFlight was added, which keeps track of the number of entries in all shards that are not currently being processed by a consumer thread. This value is updated when an entry is removed from a shard (onSuccess or onFailure) and when it's added back into the retry queue after failing processing for some reason. It's also decremented whenever we get more records out for processing - this way we can keep track of how many unprocessed records there are at any given time without having to iterate over every single record in each shard as before (which could be very expensive).
• In order to avoid starvation, if partition blocking occurs while getting work from shards, then stop scanning through them immediately instead of continuing until reaching resume point - because even though other partitions may have available messages ready for consumption right now, they will never be able to process those messages due to blocked partitions preventing progress with offset commits etc., so no need wasting CPU cycles checking these non-blocked but still unavailable message repeatedly just because they're next up according to our iteration logic based on last consumed offsets per topic/partition combination..
• Added logging around why certain WorkContainers aren't taken as work yet: delay hasn't passed; already succeeded; already failed and waiting retry period; or simply isn't allowed since its partition has been revoked temporarily due too much lag behind committed offsets etc.. Also log slow queues separately using rate limiting mechanism so only one warning gets logged every 5 seconds max about same issue(s), otherwise logs would fill up quickly with repeated warnings about same thing happening again and again...
• The testLargeVolumeInMemory() method was changed to use a different number of records.
• A new function called getTotalSizeOfAllShards() was added in the WorkManager class, which returns the total size of all shards not currently being processed by any consumer instance (i.e., it is waiting for selection). This value is used as part of an if statement that determines whether or not more work should be downloaded from Kafka and put into memory before processing begins on existing data already in memory.
• Added a new method to KafkaTestUtils, which allows the user to add records directly into the MockConsumer
• Updated all tests that use generateRecords() in order for them not to fail due to ordering issues with offsets and partitions (see note on deprecated methods)
• Fixed some bugs where we were using incorrect types of collections when adding/removing elements from lists or maps - this was causing test failures as well as potential race conditions if used in production code
• Removed unused imports and variables throughout project files
• Added @SneakyThrows to JStreamParallelEoSStreamProcessorTest.java
• Removed @beforeeach from ParallelEoSStreamProcessorTest.java
• Changed ConcurrentLinkedQueue<>() to new ConcurrentLinkedDeque<>() in OffsetEncodingBackPressureUnitTest and OffsetEncodingBackPressureUnitTest
• Replaced assertThat with ManagedTruth assertions for better error messages when the test fails (eos-test)
• Added a new test to check the final rate of progress bar is at least some value.
• Changed an assertion in WorkManagerTest class from numberOfWorkQueuedInShardsAwaitingSelection() to totalShardEntriesNotInFlight().
• Enabled debug logging for DynamicLoadFactor and ParallelEoSStreamProcessorTest classes by removing comments before logger statements in logback-test file.

[astubbs]

notes

[astubbs]

remove

[astubbs]

...

[astubbs]

lgtm

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 3 committers have signed the CLA.

✅ acktsap
❌ astubbs
❌ nachomdo
_{You have signed the CLA already but the status is still pending? Let us recheck it.}