[BUG] org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1 is flaky

[dblock]

Describe the bug

org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1 is flaky

#10256 (comment)

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1" -Dtests.seed=52E42E635A4E8817 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-QA -Dtests.timezone=Atlantic/Azores -Druntime.java=20

org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests > testRetrieveShallowCopySnapshotCase1 FAILED
    java.lang.AssertionError: 
    Expected: is <SUCCESS>
         but: was <PARTIAL>
        at __randomizedtesting.SeedInfo.seed([52E42E635A4E8817:D84FA7EB206944B4]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.repositories.blobstore.BlobStoreRepositoryHelperTests.createSnapshot(BlobStoreRepositoryHelperTests.java:142)
        at org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1(BlobStoreRepositoryRemoteIndexTests.java:156)

[harishbhakuni]

this test is passing with the given seed now:

./gradlew ':server:test' --tests "org.opensearch.repositories.blobstore.BlobStoreRepositoryRemoteIndexTests.testRetrieveShallowCopySnapshotCase1" -Dtests.seed=52E42E635A4E8817

> Configure project :
========================= WARNING =========================
         Backwards compatibility tests are disabled!
See https://github.com/opensearch-project/OpenSearch/issues/4173
===========================================================
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.5
  OS Info               : Mac OS X 13.6.3 (aarch64)
  JDK Version           : 18 (Amazon Corretto JDK)
  JAVA_HOME             : /Library/Java/JavaVirtualMachines/amazon-corretto-18.jdk/Contents/Home
  Random Testing Seed   : 52E42E635A4E8817
  In FIPS 140 mode      : false
=======================================

> Task :server:test
Jan 22, 2024 10:05:28 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.BootstrapForTesting (file:/Volumes/workplace/os-fork/OpenSearch/test/framework/build/distributions/framework-3.0.0-SNAPSHOT.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.BootstrapForTesting
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.gradle.api.internal.tasks.testing.worker.TestWorker (file:/Users/hbhakuni/.gradle/wrapper/dists/gradle-8.5-all/3zlzzgtsutfj0pbojr50n2l7z/gradle-8.5/lib/plugins/gradle-testing-base-8.5.jar)
WARNING: Please consider reporting this to the maintainers of org.gradle.api.internal.tasks.testing.worker.TestWorker
WARNING: System::setSecurityManager will be removed in a future release

this is also same issue fixed as part of #10217 .
also, ran it locally 100+ times without any failure.

[shwetathareja]

Test failed again in backport PR here - https://build.ci.opensearch.org/job/gradle-check/33283/

[peternied]

[Triage - attendees 1 2 3 4 5]
@dblock Thanks for filing this issue.

[linuxpi]

Cause of the failure in above PR

[2024-06-04T20:24:17,031][INFO ][o.o.r.b.BlobStoreRepositoryRemoteIndexTests] [testRetrieveShallowCopySnapshotCase1] --> create remote index shallow snapshot
[2024-06-04T20:24:17,031][INFO ][o.o.r.RepositoriesService] [node_s_0] update repository [test-repo]
[2024-06-04T20:24:17,259][INFO ][o.o.r.b.BlobStoreRepositoryRemoteIndexTests] [testRetrieveShallowCopySnapshotCase1] --> creating snapshot [test-snap-2] of [test-idx, test-rs-idx] in [test-repo]
[2024-06-04T20:24:17,270][INFO ][o.o.s.SnapshotsService   ] [node_s_0] snapshot [test-repo:test-snap-2/RhT560OOQpy6gC62jXfrdQ] started
[2024-06-04T20:24:17,746][WARN ][o.o.s.SnapshotShardsService] [node_s_0] Exception while acquiring lock on primaryTerm = 1 and generation = 4
[2024-06-04T20:24:18,228][WARN ][o.o.s.SnapshotShardsService] [node_s_0] [[test-idx][0]][test-repo:test-snap-2/RhT560OOQpy6gC62jXfrdQ] failed to snapshot shard
java.nio.file.NoSuchFileException: Metadata file is not present for given primary term 1 and generation 5
	at org.opensearch.index.store.RemoteSegmentStoreDirectory.getMetadataFileForCommit(RemoteSegmentStoreDirectory.java:534) ~[main/:?]
	at org.opensearch.index.store.RemoteSegmentStoreDirectory.acquireLock(RemoteSegmentStoreDirectory.java:487) ~[main/:?]
	at org.opensearch.index.shard.IndexShard.acquireLockOnCommitData(IndexShard.java:1658) ~[main/:?]
	at org.opensearch.snapshots.SnapshotShardsService.snapshot(SnapshotShardsService.java:424) [main/:?]
	at org.opensearch.snapshots.SnapshotShardsService.lambda$startNewShards$1(SnapshotShardsService.java:284) [main/:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [main/:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

[linuxpi]

The above failure is different from the original one for which the issue was opened