Pooling allocator: add a reuse-affinity policy.

[cfallin]

(Builds on #3697.)

This policy attempts to reuse the same instance slot for subsequent
instantiations of the same module. This is particularly useful when
using a pooling backend such as memfd that benefits from this reuse: for
example, in the memfd case, instantiating the same module into the same
slot allows us to avoid several calls to mmap() because the same
mappings can be reused.

The policy tracks a freelist per "compiled module ID", and when
allocating a slot for an instance, tries these two options in order:

1. A slot from the freelist for this module (i.e., last used for another
   instantiation of this particular module), or
2. A slot that was last used by some other module or never before.

The "victim" slot for choice 2 is randomly chosen.

The data structures are carefully designed so that all updates are O(1),
and there is no retry-loop in any of the random selection.

This policy is now the default when the memfd backend is selected via
the memfd-allocator feature flag.

[cfallin]

After writing this, it occurs to me that the reuse policy as stated will choose with equal probability a module whose freelist we steal from, but this does not imply equal probability for any slot to be stolen.

In other words, if we have one module with average occupancy of 500 preinitialized slots out of 1000, and 500 others with 1 slot each, and a new module comes along and wants a slot, we have only 1/501 chance of picking one of the 500.

To unbias this I should probably keep a global freelist (whole pool of choices mixed together), randomly pick from that freelist, and keep a reverse-index of slot to last-allocated module to remove the index from that module's freelist (or lazily do so next time we look at that list). I'll take a closer look at this tomorrow!

[github-actions]

Subscribe to Label Action

cc @peterhuene

This issue or pull request has been labeled: "wasmtime:api"

Thus the following users have been cc'd because of the following labels:

• peterhuene: wasmtime:api

To subscribe or unsubscribe from this label, edit the .github/subscribe-to-label.json configuration file.

Learn more.

[cfallin]

I've updated this PR now to use a better data structure/algorithm design. It performs a fair random choice of victim slot when no slots with the desired affinity are available, and it has all O(1) updates -- somewhat tricky given the need to maintain two freelists (global and per-module) and remove from both. This is done by keeping Vecs and using swap_remove, and tracking a slot's position in each freelist in a separate reverse-index. Hopefully the comments make this a little more clear.

I've added a randomized test that counts ID-reuse (a little random simulation of sorts) and verifies a reasonable hit rate (at least twice what would be expected with random reuse) as well.

[fitzgen]

The policy tracks a freelist per "compiled module ID", and when
allocating a slot for an instance, tries these three options in order:

1. A slot from the freelist for this module (i.e., last used for another
  instantiation of this particular module), or
3. A slot that was last used by some other module or never before.

1..3 👀

[cfallin]

1..3 👀

Incomplete edit, sorry! The distinction between the last two (empty, and then slot with other affinity) was removed because it made the data structure simpler, and in steady-state (past the first n_slot instantiations in the process) no slots will be empty.

[fitzgen]

This is looking really good! A couple questions, suggestions, and nitpicks below.

[cfallin]

I think I addressed all your comments; thanks @fitzgen ! This is rebased on the latest #3697 as well.

[fitzgen]

Thanks! (Note that I haven't looked at any of the earlier commits)