[BUG] org.opensearch.test.telemetry.tracing.validators.AllSpansHaveUniqueId fails with java.lang.OutOfMemoryError sporadically

[reta]

Describe the bug

The org.opensearch.test.telemetry.tracing.validators.AllSpansHaveUniqueId fails with java.lang.OutOfMemoryError sporadically:

java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([11E79186241F3EB2:AD561042C55D1621]:0)
	at java.base/java.util.HashMap.newNode(HashMap.java:1909)
	at java.base/java.util.HashMap.putVal(HashMap.java:637)
	at java.base/java.util.HashMap.put(HashMap.java:618)
	at java.base/java.util.HashSet.add(HashSet.java:229)
	at org.opensearch.test.telemetry.tracing.validators.AllSpansHaveUniqueId.validate(AllSpansHaveUniqueId.java:42)
	at org.opensearch.test.telemetry.tracing.TelemetryValidators.validate(TelemetryValidators.java:37)
	at org.opensearch.test.telemetry.tracing.StrictCheckSpanProcessor.validateTracingStateOnShutdown(StrictCheckSpanProcessor.java:79)
	at org.opensearch.test.OpenSearchTestClusterRule.afterClass(OpenSearchTestClusterRule.java:99)
	at org.opensearch.test.OpenSearchTestClusterRule.initializeSuiteScope(OpenSearchTestClusterRule.java:399)
	at org.opensearch.test.OpenSearchTestClusterRule.before(OpenSearchTestClusterRule.java:168)
	at org.opensearch.test.OpenSearchTestClusterRule$1.evaluate(OpenSearchTestClusterRule.java:365)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)

Related component

Other

To Reproduce

Not easy to reproduce by see please:

• https://build.ci.opensearch.org/job/gradle-check/34686/testReport/junit/org.opensearch.search.aggregations.metrics/ExtendedStatsIT/testScriptSingleValued__p0___search_concurrent_segment_search_enabled___true___/

Expected behavior

No java.lang.OutOfMemoryError should be raised

Additional Details

Plugins
Standard

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/34686/testReport/junit/org.opensearch.search.aggregations.metrics/ExtendedStatsIT/testScriptSingleValued__p0___search_concurrent_segment_search_enabled___true___/
• https://build.ci.opensearch.org/job/gradle-check/33741/testReport/junit/org.opensearch.search.aggregations.metrics/GeoCentroidIT/testMultiValuedField__p0___search_concurrent_segment_search_enabled___true___/

[reta]

@Gaganjuneja please take a look at your convenience, thank you

[Gaganjuneja]

@Gaganjuneja please take a look at your convenience, thank you

Sure, thanks!

[mch2]

I think this is causing flakiness across the board ex - #12801 and other symptoms slow network thread... in tests.

In the linked test i could see up to 100k spans passed to this validate method.

[mch2]

Have raised #12877 to disable this plugin until this is resolved.

Looks like these spans are added for at least every write/replication request and not cleared until after the test executes.

[mch2]

Related - #12877 (comment)
@Gaganjuneja, Do you know why we don't see any telemetry test failures with this disabled?

[Gaganjuneja]

This is to verify if all the spans are closed. Telemetry tests are not dependent on this feature.