Avoid False Positives 409s in Thanos Receive Replication

[squat]

There is a little bit of nuance in the replication error logic that we can improve in order to avoid false positives when discovering 409s. Namely, when the replicate func checks for errors, it should only return the 409 if the number of 409s in the multi-error was greater than or equal to the minimum replication threshold. Otherwise, the replication should be retried.

In order words, let's say we have replication factor 3. This means the minimum replication threshold is 2.

• if there was only 1 error, then no matter what the error was, the replication will succeed and return a nil error;
• if there were two or more errors:
  • if just one of the errors was a 409, then we shouldn't return a 409 because this didn't really cause the failure, the 500 from the other error is responsible; the error should either be fixed in code, or prometheus should retry sending the sample;
  • if two or more errors were 409s, then the sample is definitely conflicting and should not be retried. There is no internal server error that could be fixed and so retrying will never work

xref: #1563 #1586