Create memory queue

[mariusandra]

For job queues we currently have (almost) two implemented: 1) graphile, 2) S3

We could add a third one: memory. This should only work with runNow() jobs and only if there are not too many of them.

The tricky part is that jobs are enqueued in worker threads, but are then distributed to workers in the main thread. Thus we need some form of communication channel between threads. Jobs can be huge (100MB for S3?), so support for buffering/chunking the payload when sending between threads might be needed.

The GeoIP plugin implements a local TCP server for communication between threads. Perhaps this could be used as well here, or some other channel can be created.

We're running a patched fork of Piscina and they didn't want to accept our changes upstream, so we can go wild with how we implement this :).

Also, this memory queue should

1. have some way of knowing "sorry, I'm full" and then rejecting jobs
2. have some way of flushing all events to the next persistent queue (postgres probably) when the server terminates

[posthog-bot]

This issue hasn't seen activity in two years! If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[posthog-bot]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.