Fix SegRep bug where primary shards do not update tracked replica state post failover/relocation

[mch2]

Description

This change fixes a bug that can occur with SegRep where primary shard does not update its local tracking of replica ReplicationCheckpoint state. This can occur post failover or relocation where a new primary publishes a checkpoint that is swallowed by replicas that are already up to date. Unless a new write is made to the index, the lag will grow indefinitely.

Related Issues

Resolves #11016

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 8930681

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadRangeBlobWithRetries

• URL: https://build.ci.opensearch.org/job/gradle-check/29341/
• CommitID: 8930681
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #11017 (8930681) into main (63aff16) will increase coverage by 0.05%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main   #11017      +/-   ##
============================================
+ Coverage     71.29%   71.34%   +0.05%     
- Complexity    58742    58824      +82     
============================================
  Files          4872     4872              
  Lines        276777   276780       +3     
  Branches      40240    40241       +1     
============================================
+ Hits         197316   197468     +152     
+ Misses        62943    62851      -92     
+ Partials      16518    16461      -57     

Files	Coverage Δ
...s/replication/SegmentReplicationTargetService.java	56.17% <100.00%> (+0.56%)	⬆️

... and 486 files with indirect coverage changes