fix(bitswap/client/msgq): prevent duplicate requests

[Wondertan]

Previously, in-progress requests could be re-requested again during periodic rebroadcast. The queue requests, and while awaiting a response, the rebroadcast event happens. Rebroadcast event changes previously sent WANTs to pending and sends them again in a new message, duplicating some WANT requests.

The solution here is to ensure WANT was in sent status for long enough before bringing it back to pending. This utilizes the existing sendAt map, which tracks when every CID was sent. Then, on every event, it compares if the message was around longer than rebroadcastInterval

[Wondertan]

Alternatively, this could be a different new parameter/constant

[Wondertan]

I tested this on a k8s cluster and with a local node connected to it. It works as expected, but I believe this would benefit a lot from a proper test. Unfortunately, I can't allocate time to writing one. It's not that straightforward.

[Wondertan]

For context, I detect duplicates with a custom multihash that logs out when the same data is hashed again. This essentially uncovered #690, and this issue

[codecov]

Codecov Report

Attention: Patch coverage is 96.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 60.39%. Comparing base (37756ce) to head (d67691d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...tswap/client/internal/messagequeue/messagequeue.go	96.00%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main     #691      +/-   ##
==========================================
+ Coverage   60.36%   60.39%   +0.03%     
==========================================
  Files         244      244              
  Lines       31034    31044      +10     
==========================================
+ Hits        18734    18750      +16     
+ Misses      10630    10626       -4     
+ Partials     1670     1668       -2     

Files with missing lines	Coverage Δ
bitswap/client/wantlist/wantlist.go	90.90% <ø> (-0.88%)	⬇️
...tswap/client/internal/messagequeue/messagequeue.go	84.06% <96.00%> (+0.53%)	⬆️

... and 13 files with indirect coverage changes

[gammazero]

This is probably good to leave since it avoids Lock/Unlock of mq.rebroadcastIntervalLk and time.Now().

	if mq.bcstWants.sent.Len() == 0 && mq.peerWants.sent.Len() == 0 {
		return 0
	}

[Wondertan]

The lock exists only for testing. The interval is never changed outside of the unit test. Thus, I don't see any contention zero length check could prevent.

[hsanjuan]

I think the comment is not about contention but about saving unnecessary lock/unlock calls, but if this only happens every 30 seconds, then it's probably not very important.

[gammazero]

triage note: This is a good candidate for testing in rainbow staging to observe performance differences.

[hsanjuan]

The main thing to consider here is that:

• before, a "want" would be re-broadcasted at most 30 seconds after it was sent (could be 0.1s)
• after, a "want" would be re-broadcasted only after at least 30 seconds after it was sent (could be 59.9s).

In that respect the code looks good.

I am not sure how much of an improvement this is in practice (perhaps clients were lucky to hit a short rebroadcast period sometimes), but it makes clients more respectful at least and perf should not be based on "luck".

I think we can test on staging and discuss in the next triage if we accept the change.

[hsanjuan]

I think the comment is not about contention but about saving unnecessary lock/unlock calls, but if this only happens every 30 seconds, then it's probably not very important.

[gammazero]

Need to test on staging before merge.

[gammazero]

This PR does make sure that the client does not resend wants to any peer before the rebroadcast interval has elapsed. In doing this is also makes some peers, that were just short of the interval, wait for another rebroadcast interval.

In summary it changes from "wait no more than X to resend wants" to "wait at least X, but no more than 2X, to resend wants".

If we want the PR, then we should consider calling rebroadcastWantlist at half (or less) of the rebroadcast interval. @hsanjuan WDYT?

Consider changing line 410 to

const checksPerInterval = 2
mq.rebroadcastTimer = mq.clock.Timer(mq.rebroadcastInterval / checksPerInterval)

That will change the logic to "wait at least X, but no more than X+(X/checksPerInterval), to resend wants