feat: Improve content propagation in sync

[Frando]

Description

This improves content propagation for document sync by implementing the following features.

• feat: allow to send gossip to neighbors only
• feat: provider and candidate roles for downloader
• feat: transfer content status during set reconciliation
• feat: inform neighbors about finished content downloads

Notes & open questions

Change checklist

• Self-review.
• Documentation updates if relevant.
• Tests if relevant.

[Frando]

sync_full_basic times out. I think it's due to how I changed the downloader, which is incorrect: I changed the delay for request scheduling to 0 in the case of peers marked as Provider. However when I'm rereading the code now, I think I'm noticing that this delay currently is intended to cover also the time it takes to establish the connection. So it fails, because the connection is not established in zero time. @divagant-martian maybe you can have a look at the commit that changes the downloader, maybe you have a good idea on how to wire this correctly without having to change too much of the downloader logic.

[divagant-martian]

upon receiving the request you can only start it right away if the provider is connected. what we need to do is, when a provider has capacity (which includes ofc when it first connects) to get the next hash if provides eagerly, so changes there are a bit more involved. My plan was to store this in the ProviderMap as a HashMap<PublicKey, VecDeque<Hash>> everytime a hash is tried for this peer the Hash is moved to the back of the queue, so that we try different hashes for a single provider. A peer from the map would be removed when all it's provided hashes are removed, and hashes are removed when downloads are done.

In this way we combine needing to have delays on retries and eager downloads from known providers. I know you would like to consider delays per (hash, peer) but I would wait until we see if this is necessary since that's more overhead than what's potentially necessary to effectively get a download.

Also, I think it's a reasonable option to limit the PR up to where we define the peer priority and handle eager downloads in another one since what's in here is a good improvement per se already. This way you are not blocked by the changes of the downloader in the meantime

[Frando]

I added two commits:

• make the downloader start downloading hashes added with PeerRole:::Provider immediately. Implements @divagant-martian's suggestion from the previous commit.
• Expose the distance (in hops) that a gossip message travelled. Use this to treat sync gossip messages with a distance of 0 (i.e. received from author directly) to use PeerRole::Provider (i.e. assume that the author has the content available).

With these two changes, tests should now pass! And all artificial delays for content downloads in the happy case are gone now.

I think this is ready for review now. Noted some other things we might do to improve sync, but those should go into new PRs.

[divagant-martian]

just starting here

[divagant-martian]

Looks good to me! I can't vouch more for the sync related code but gossip and downloads look generally good. Comments are minor

[Frando]

Thanks for the review. Addressed the comments, let's see what CI says :)

[divagant-martian]

lgtm!