Add random delay to ArchiveExecutionTask

[MichaelSnowden]

What changed?

1. I made the archival queue a scheduled queue instead of a timer queue.
2. I added a random delay to each task we produce

Why?
I did this to prevent us from having a ton of archive execution tasks arrive concurrently if many workflows close simultaneously.

How did you test it?
I added unit tests that verify edge cases like setting the dynamic config upper bound to 0.

Potential risks
Does not run via any production entrypoints.

Is hotfix candidate?
No.

[yycptt]

Archival queue is an immediate queue, there's no delay by definition & task visibility timestamp for immediate tasks (which is only for metrics purpose) will be overwritten to time.Now() by shard context.

[yux0]

why do it need logger here?

[MichaelSnowden]

Probably from an older revision. Removed, thanks for spotting!

[yux0]

backoff.JitDuration(delay, 1) / 2

[MichaelSnowden]

Fixed, thanks!

[yux0]

Archival queue is an immediate queue, there's no delay by definition & task visibility timestamp for immediate tasks (which is only for metrics purpose) will be overwritten to time.Now() by shard context.

With task processing rate limiter, do we still need the jitter here? I think this is the main reason we want to change it to scheduled queue.

[yycptt]

Is this used? Remove?

[MichaelSnowden]

Removed, thanks!

[yycptt]

Archival queue is an immediate queue, there's no delay by definition & task visibility timestamp for immediate tasks (which is only for metrics purpose) will be overwritten to time.Now() by shard context.

With task processing rate limiter, do we still need the jitter here? I think this is the main reason we want to change it to scheduled queue.

If we can avoid the spiky load in the first place, I think it's better than relying on the rate limiter to handle it after it occurs, which exists a possibility that the rate limiter can't handle it well (especially for tasks that needs to read history.) Plus, I don't think scheduled queue is a lot more expensive than immediate queues (especially when load is high).