WatchBackwardsCompatibilityIT failure

[dimitris-athanasiou]

Build link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/190/console

Reproduce:

./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.3.1#oldClusterTestRunner -Dtests.seed=7151E0B6D1D9F670 -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT -Dtests.method="testWatchCrudApis" -Dtests.security.manager=true -Dtests.locale=es-CU -Dtests.timezone=Etc/GMT-3 -Dcompiler.java=12 -Druntime.java=8

Failure:

ERROR   0.81s | WatchBackwardsCompatibilityIT.testWatchCrudApis <<< FAILURES!
   > Throwable #1: org.elasticsearch.client.ResponseException: method [PUT], host [http://127.0.0.1:45421], URI [/.triggered_watches], status line [HTTP/1.1 400 Bad Request]
   > {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [.triggered_watches/cOEVmRIeSA2jjqs67I1PCA] already exists","index_uuid":"cOEVmRIeSA2jjqs67I1PCA","index":".triggered_watches"}],"type":"resource_already_exists_exception","reason":"index [.triggered_watches/cOEVmRIeSA2jjqs67I1PCA] already exists","index_uuid":"cOEVmRIeSA2jjqs67I1PCA","index":".triggered_watches"},"status":400}
   > 	at __randomizedtesting.SeedInfo.seed([7151E0B6D1D9F670:2BE982E89CCA769E]:0)
   > 	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936)
   > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
   > 	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:278)
   > 	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$waitForSecuritySetup$1(WatchBackwardsCompatibilityIT.java:131)
   > 	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:272)
   > 	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.waitForSecuritySetup(WatchBackwardsCompatibilityIT.java:118)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://127.0.0.1:45421], URI [/.triggered_watches], status line [HTTP/1.1 400 Bad Request]
   > {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [.triggered_watches/cOEVmRIeSA2jjqs67I1PCA] already exists","index_uuid":"cOEVmRIeSA2jjqs67I1PCA","index":".triggered_watches"}],"type":"resource_already_exists_exception","reason":"index [.triggered_watches/cOEVmRIeSA2jjqs67I1PCA] already exists","index_uuid":"cOEVmRIeSA2jjqs67I1PCA","index":".triggered_watches"},"status":400}
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:552)
   > 	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
   > 	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
   > 	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
   > 	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
   > 	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   > 	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   > 	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   > 	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   > 	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   > 	at org.apache.http.imp

[elasticmachine]

Pinging @elastic/es-core-features

[davidkyle]

This failure is a different test in the suite but the error is the same so I'm adding here given both errors come from WatchBackwardsCompatibilityIT::waitForSecuritySetup

org.elasticsearch.client.ResponseException: method [PUT], host [http://127.0.0.1:34573], URI [/.triggered_watches], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [.triggered_watches/D_ji91T6Qbyou79tnQtLzg] already exists","index_uuid":"D_ji91T6Qbyou79tnQtLzg","index":".triggered_watches"}],"type":"resource_already_exists_exception","reason":"index [.triggered_watches/D_ji91T6Qbyou79tnQtLzg] already exists","index_uuid":"D_ji91T6Qbyou79tnQtLzg","index":".triggered_watches"},"status":400}
	at __randomizedtesting.SeedInfo.seed([F8F968D470C9721F:84CF713E8285636B]:0)
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:278)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$waitForSecuritySetup$1(WatchBackwardsCompatibilityIT.java:131)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:272)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.waitForSecuritySetup(WatchBackwardsCompatibilityIT.java:118)

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+bwc-tests/213/console

./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.4.2#oldClusterTestRunner \
  -Dtests.seed=F8F968D470C9721F \
  -Dtests.class=org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT \
  -Dtests.method="testWatcherStats" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-RS \
  -Dtests.timezone=Africa/Algiers \
  -Dcompiler.java=12 \
  -Druntime.java=8

[dimitris-athanasiou]

Another failure same as the one in the issue description:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+default-distro+bwc/BWC_VERSION=6.4.0,nodes=general-purpose/3/console

[davidkyle]

Another

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+default-distro+bwc/BWC_VERSION=6.0.1,nodes=centos-7&&immutable/43/console

org.elasticsearch.client.ResponseException: method [PUT], host [http://127.0.0.1:33117], URI [/.triggered_watches], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [.triggered_watches/PkRq2oFaRGm2gYnweXeBRA] already exists","index_uuid":"PkRq2oFaRGm2gYnweXeBRA","index":".triggered_watches"}],"type":"resource_already_exists_exception","reason":"index [.triggered_watches/PkRq2oFaRGm2gYnweXeBRA] already exists","index_uuid":"PkRq2oFaRGm2gYnweXeBRA","index":".triggered_watches"},"status":400}
	at __randomizedtesting.SeedInfo.seed([66AEF0D76756E4AD:1A98E93D951AF5D9]:0)
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:278)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.lambda$waitForSecuritySetup$1(WatchBackwardsCompatibilityIT.java:131)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.executeAgainstMasterNode(WatchBackwardsCompatibilityIT.java:272)
	at org.elasticsearch.upgrades.WatchBackwardsCompatibilityIT.waitForSecuritySetup(WatchBackwardsCompatibilityIT.java:118)

[davidkyle]

Muted b376cbc

[jakelandis]

Closing this issue as won't fix since this has been open for a long time and there are no plans to invest the time necessary to fix this test. Multiple people have spent considerable time trying to get the Watcher tests stable in 7.x and master. Those changes include many updates to the test framework as well as a few production level changes that allow for better stop/start/stop/start in quick succession workflows (those only happen in the tests). The vast majority of test failures are due to the stop/start/stop nature and either leaving behind artifacts, or timing issues w.r.t stopping not actually being fully stopped before starting again. Since the identified issues in 7.x and master have not been real issues and the root cause of these failures is how we test, and that none of these changes to stabilize tests were back ported to 6.8 I will close this issue and leave the test muted. Additionally, this specific test is testing the upgrade from 5.x to 6.8 and there have not been any reports of issues with that upgrade and this test does not exist in 7.x+.