Plugin server gets stuck when ingesting via Kafka

[mariusandra]

We have had cases where the Kafka-powered plugin server just stops working. Such as last night on cloud.

Plugin server ingestion on its own is turned off, yet I'm still running the "queue latency" plugin, which emits one event via posthog.capture every minute. Between noon and 9am, the plugin server stopped ingesting events.

Here's what the logs show:

The line took to long to run plugins on... is part of a system to detect if and when eachBatch gets stuck (without killing anything).

What exactly happened is still to be determined. Everything came back online when the task was redeployed this morning.

[mariusandra]

This can be closed I hope. We now have timeouts and a lot of metrics while processing the Kafka batch. The "30sec timeout warning" messages will also be sent to Sentry, so we should keep an eye on them ther.e

[mariusandra]

The timeouts helped catch several bugs. The big problem though was unpooled concurrent redis requests (more here).