Idea: Add an option to send purges in batches.

[harlow]

Does Fastly support a batch purge? We've been putting purge requests into the background and leveraging sidekiq unique jobs to try to reduce outgoing HTTP traffic.

[ezkl]

We don't really have anything like a batch purge aside from surrogate key purge and purge_all. The unique job queue is definitely a cool idea. Depending on frequency of purge, I wonder if leveraging persistent HTTP connections would help as well?

[set5think]

We could queue up surrogate keys to be purged in a defined time interval, or up to a certain number of keys, and send one request out. We could additionally squash them to be unique, in case the same key shows up multiple times.

Although we do allow n surrogate keys to be passed in with any given surrogate key purge request, the nature of apps will probably only purge one or two at a time.

Thoughts?

[set5think]

We discussed some more internally - it seems the consensus is that this idea would be better served as an extension to fastly-rails. I guess the next prerequisite step to that, though, is to make fastly-rails extendable.

[harlow]

Yeah seems like a place fastly-rails could be great. Maybe we could pass the queue strategy into the initializer (Sidekiq, Resque, other)?

[ezkl]

Maybe we could pass the queue strategy into the initializer (Sidekiq, Resque, other)?

👍