Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Watcher SingleNodeTests failed due to running threads #36782

Closed
spinscale opened this issue Dec 18, 2018 · 11 comments
Closed

[CI] Watcher SingleNodeTests failed due to running threads #36782

spinscale opened this issue Dec 18, 2018 · 11 comments
Assignees
@martijnvg
Labels
:Data Management/Watcher >test-failure Triaged test failures from CI

Comments

@spinscale
Copy link
Contributor

See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+periodic/376/console

the logfile is pretty clobbered as two exceptions are written at the same time, this is the first one

  1> [2018-12-18T20:24:37,292][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] before test
  1> [2018-12-18T20:24:37,294][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] [SingleNodeTests#testThatLoadingWithNonExistingIndexWorks]: setting up test
  1> [2018-12-18T20:24:37,295][INFO ][o.e.t.InternalTestCluster] [testThatLoadingWithNonExistingIndexWorks] Setup InternalTestCluster [SUITE-CHILD_VM=[0]-CLUSTER_SEED=[-3226635383265237318]-HASH=[5B37D0D7D73]-cluster] with seed [D338B036E0C4E6BA] using [0] dedicated masters, [1] (data) nodes and [0] coord only nodes (min_master_nodes are [auto-managed])
  1> [2018-12-18T20:24:37,318][INFO ][o.e.e.NodeEnvironment    ] [testThatLoadingWithNonExistingIndexWorks] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [466.8gb], net total_space [496.2gb], types [ext4]
  1> [2018-12-18T20:24:37,318][INFO ][o.e.e.NodeEnvironment    ] [testThatLoadingWithNonExistingIndexWorks] heap size [491mb], compressed ordinary object pointers [true]
  1> [2018-12-18T20:24:37,319][INFO ][o.e.n.Node               ] [testThatLoadingWithNonExistingIndexWorks] node name [node_s0], node ID [O4v3E-nKQCy0igFZ1QkErQ]
  1> [2018-12-18T20:24:37,319][INFO ][o.e.n.Node               ] [testThatLoadingWithNonExistingIndexWorks] version[6.6.0-SNAPSHOT], pid[1510], build[unknown/unknown/35b1b60/2018-12-18T13:14:16.652553Z], OS[Linux/4.15.0-1025-gcp/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_192/25.192-b12]
  1> [2018-12-18T20:24:37,320][INFO ][o.e.n.Node               ] [testThatLoadingWithNonExistingIndexWorks] JVM arguments [-Dfile.encoding=UTF8, -ea, -esa, -Xmx512m, -Xms512m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/jenkins/workspace/elastic+elasticsearch+6.x+periodic/x-pack/plugin/watcher/build/heapdump, -Dtests.prefix=tests, -Dtests.seed=86F308EEE121B18C, -Dtests.security.manager=true, -Des.http.cname_in_publish_address=true, -Djava.io.tmpdir=./temp, -Dtests.gradle=true, -Des.set.netty.runtime.available.processors=false, -Des.aggregations.enable_scripted_metric_agg_param=false, -Dcompiler.java=11, -Dtests.task=:x-pack:plugin:watcher:test, -Djna.nosys=true, -Druntime.java=8, -Des.scripting.update.ctx_in_params=false, -Dtests.artifact=watcher, -Djava.awt.headless=true, -Dtests.logger.level=WARN, -Djunit4.childvm.cwd=/var/lib/jenkins/workspace/elastic+elasticsearch+6.x+periodic/x-pack/plugin/watcher/build/testrun/test/J0, -Djunit4.tempDir=/var/lib/jenkins/workspace/elastic+elasticsearch+6.x+periodic/x-pack/plugin/watcher/build/testrun/test/temp, -Djunit4.childvm.id=0, -Djunit4.childvm.count=8]
  1> [2018-12-18T20:24:37,320][WARN ][o.e.n.Node               ] [testThatLoadingWithNonExistingIndexWorks] version [6.6.0-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
  1> [2018-12-18T20:24:37,324][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] no modules loaded
  1> [2018-12-18T20:24:37,324][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.analysis.common.CommonAnalysisPlugin]
  1> [2018-12-18T20:24:37,324][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.script.MockMustacheScriptEngine$TestPlugin]
  1> [2018-12-18T20:24:37,324][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.test.ESIntegTestCase$AssertActionNamePlugin]
  1> [2018-12-18T20:24:37,325][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.test.ESIntegTestCase$TestSeedPlugin]
  1> [2018-12-18T20:24:37,325][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.test.discovery.TestZenDiscovery$TestPlugin]
  1> [2018-12-18T20:24:37,325][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.test.store.MockFSIndexStore$TestPlugin]
  1> [2018-12-18T20:24:37,325][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.transport.MockTcpTransportPlugin]
  1> [2018-12-18T20:24:37,325][INFO ][o.e.p.PluginsService     ] [testThatLoadingWithNonExistingIndexWorks] loaded plugin [org.elasticsearch.xpack.watcher.test.LocalStateWatcher]
  1> [2018-12-18T20:24:37,439][INFO ][o.e.d.DiscoveryModule    ] [testThatLoadingWithNonExistingIndexWorks] using discovery type [test-zen] and host providers [settings, file]
  1> [2018-12-18T20:24:37,508][INFO ][o.e.n.Node               ] [testThatLoadingWithNonExistingIndexWorks] initialized
  1> [2018-12-18T20:24:37,525][INFO ][o.e.n.Node               ] [integ_1] starting ...
  1> [2018-12-18T20:24:37,541][INFO ][o.e.t.TransportService   ] [integ_1] publish_address {127.0.0.1:40335}, bound_addresses {[::1]:33335}, {127.0.0.1:40335}
  1> [2018-12-18T20:24:37,563][INFO ][o.e.t.d.MockZenPing      ] [node_s0] pinging using mock zen ping
  1> [2018-12-18T20:24:37,593][INFO ][o.e.c.s.MasterService    ] [node_s0] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {node_s0}{O4v3E-nKQCy0igFZ1QkErQ}{KCchiGFwTrCutDk-9uGtow}{127.0.0.1}{127.0.0.1:40335}{xpack.installed=true}
  1> [2018-12-18T20:24:37,594][INFO ][o.e.c.s.ClusterApplierService] [node_s0] new_master {node_s0}{O4v3E-nKQCy0igFZ1QkErQ}{KCchiGFwTrCutDk-9uGtow}{127.0.0.1}{127.0.0.1:40335}{xpack.installed=true}, reason: apply cluster state (from master [master {node_s0}{O4v3E-nKQCy0igFZ1QkErQ}{KCchiGFwTrCutDk-9uGtow}{127.0.0.1}{127.0.0.1:40335}{xpack.installed=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
  1> [2018-12-18T20:24:37,624][INFO ][o.e.n.Node               ] [integ_1] started
  1> [2018-12-18T20:24:37,669][INFO ][o.e.g.GatewayService     ] [node_s0] recovered [0] indices into cluster_state
  1> [2018-12-18T20:24:37,844][INFO ][o.e.l.LicenseService     ] [node_s0] license [3f17f028-e1ed-4372-a71b-2bb9aa0fa567] mode [trial] - valid
  1> [2018-12-18T20:24:37,849][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [.watches] for index patterns [.watches*]
  1> [2018-12-18T20:24:37,863][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
  1> [2018-12-18T20:24:37,877][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [.triggered_watches] for index patterns [.triggered_watches*]
  1> [2018-12-18T20:24:37,884][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] adding template [random_index_template] for index patterns [*]
  1> [2018-12-18T20:24:37,888][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] [SingleNodeTests#testThatLoadingWithNonExistingIndexWorks]: all set up test
  1> [2018-12-18T20:24:37,896][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [.watches] creating index, cause [api], templates [.watches, random_index_template], shards [3]/[0], mappings [doc]
  1> [2018-12-18T20:24:37,982][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.watches][1]] ...]).
  1> [2018-12-18T20:24:37,997][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [.triggered_watches] creating index, cause [api], templates [.triggered_watches, random_index_template], shards [1]/[0], mappings [doc]
  1> [2018-12-18T20:24:38,050][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.triggered_watches][0]] ...]).
  1> [2018-12-18T20:24:38,068][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [.watcher-history-9-2018.12.18] creating index, cause [api], templates [.watch-history-9, random_index_template], shards [1]/[0], mappings [doc]
  1> [2018-12-18T20:24:38,126][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.watcher-history-9-2018.12.18][0]] ...]).
  1> [2018-12-18T20:24:38,130][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] creating watch history index [.watcher-history-9-2018.12.18]
  1> [2018-12-18T20:24:38,132][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STARTED]]
  1> [2018-12-18T20:24:38,132][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] waiting to stop watcher, current states [Tuple [v1=node_s0, v2=STARTED]]
  1> [2018-12-18T20:24:38,144][INFO ][o.e.x.w.WatcherService   ] [node_s0] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2018-12-18T20:24:38,147][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] waiting to stop watcher, current states [Tuple [v1=node_s0, v2=STOPPED]]
  1> [2018-12-18T20:24:38,149][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [.watches/N3TThpZqTpuHDHmW7s7yIQ] deleting index
  1> [2018-12-18T20:24:38,178][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STOPPED]]
  1> [2018-12-18T20:24:38,184][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] waiting to start watcher, current states [Tuple [v1=node_s0, v2=STARTED]]
  1> [2018-12-18T20:24:38,190][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0] [.watches] creating index, cause [auto(bulk api)], templates [.watches, random_index_template], shards [1]/[0], mappings [doc]
  1> [2018-12-18T20:24:38,230][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.watches][0]] ...]).
  1> [2018-12-18T20:24:38,237][INFO ][o.e.x.w.WatcherService   ] [node_s0] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
  1> [2018-12-18T20:24:39,506][INFO ][o.e.x.w.a.l.ExecutableLoggingAction] [node_s0] logging of watch _name
  1> [2018-12-18T20:24:39,605][INFO ][o.e.c.m.MetaDataMappingService] [node_s0] [.watcher-history-9-2018.12.18/pSeFwyWESYi6DH8TfpW46A] update_mapping [doc]
  1> [2018-12-18T20:24:40,497][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] [#testThatLoadingWithNonExistingIndexWorks]: clearing watcher state
  1> [2018-12-18T20:24:40,498][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] waiting to stop watcher, current states [Tuple [v1=node_s0, v2=STARTED]]
  1> [2018-12-18T20:24:40,502][INFO ][o.e.x.w.WatcherService   ] [node_s0] stopping watch service, reason [watcher manually marked to shutdown by cluster state update]
  1> [2018-12-18T20:24:40,517][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] waiting to stop watcher, current states [Tuple [v1=node_s0, v2=STOPPED]]
  1> [2018-12-18T20:24:40,517][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] [SingleNodeTests#testThatLoadingWithNonExistingIndexWorks]: cleaning up after test
  1> [2018-12-18T20:24:40,532][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [.watches/_8IMNfj5RkOM6yb3gOxNEA] deleting index
  1> [2018-12-18T20:24:40,532][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [.triggered_watches/rfGF8WCbScit-jHzFztw3g] deleting index
  1> [2018-12-18T20:24:40,532][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0] [.watcher-history-9-2018.12.18/pSeFwyWESYi6DH8TfpW46A] deleting index
  1> [2018-12-18T20:24:40,586][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node_s0] removing template [random_index_template]
  1> [2018-12-18T20:24:40,590][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] [SingleNodeTests#testThatLoadingWithNonExistingIndexWorks]: cleaned up after test
  1> [2018-12-18T20:24:40,590][INFO ][o.e.x.w.t.i.SingleNodeTests] [testThatLoadingWithNonExistingIndexWorks] after test
  1> [2018-12-18T20:24:40,591][INFO ][o.e.n.Node               ] [suite] stopping ...
  1> [2018-12-18T20:24:40,591][INFO ][o.e.x.w.WatcherService   ] [suite] stopping watch service, reason [shutdown initiated]
  1> [2018-12-18T20:24:40,997][WARN ][o.e.x.w.e.ExecutionService] [node_s0] failed to execute watch [zTGvxQgYlWQUBLACsUlx]
  1> [2018-12-18T20:24:40,998][INFO ][o.e.n.Node               ] [suite] stopped
  1> [2018-12-18T20:24:40,999][INFO ][o.e.n.Node               ] [suite] closing ...
  1> [2018-12-18T20:44:42,418][INFO ][o.e.a.b.BulkRequestHandler] [node_s0] Bulk request 3 has been cancelled.
  1> java.lang.InterruptedException: null
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_192]
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_192]
  1> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) ~[?:1.8.0_192]
  1> 	at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:85) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:374) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:365) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:322) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:305) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:301) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:287) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.watcher.history.HistoryStore.put(HistoryStore.java:49) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:314) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:394) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:543) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2018-12-18T20:44:42,419][ERROR][o.e.x.w.Watcher          ] [node_s0] error executing bulk
  1> java.lang.InterruptedException: null
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998) ~[?:1.8.0_192]
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_192]
  1> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) ~[?:1.8.0_192]
  1> 	at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:85) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:374) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:365) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:322) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:305) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:301) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:287) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.watcher.history.HistoryStore.put(HistoryStore.java:49) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:314) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:394) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:543) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2018-12-18T20:44:42,420][INFO ][o.e.a.b.BulkRequestHandler] [node_s0] Bulk request 4 has been cancelled.
  1> java.lang.InterruptedException: null
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) ~[?:1.8.0_192]
  1> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) ~[?:1.8.0_192]
  1> 	at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:59) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:374) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:365) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:322) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:305) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:301) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:294) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.watcher.execution.TriggeredWatchStore.delete(TriggeredWatchStore.java:116) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:322) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:394) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:543) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2018-12-18T20:44:42,421][ERROR][o.e.x.w.Watcher          ] [node_s0] error executing bulk
  1> java.lang.InterruptedException: null
  1> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) ~[?:1.8.0_192]
  1> 	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) ~[?:1.8.0_192]
  1> 	at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:59) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:374) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:365) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:322) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:305) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:301) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:294) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.watcher.execution.TriggeredWatchStore.delete(TriggeredWatchStore.java:116) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:322) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:394) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:543) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_192]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_192]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
  1> [2018-12-18T20:44:42,424][INFO ][o.e.n.Node               ] [suite] closed

this is the second one containing jstack output

  2> dec. 18, 2018 8:44:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.elasticsearch.xpack.watcher.test.integration.SingleNodeTests
  2> ==== jstack at approximately timeout time ====
  2> "elasticsearch[node_s0][management][T#2]" ID=180 TIMED_WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6b0359e4
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6b0359e4
  2> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1277)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][index][T#2]" ID=179 WAITING on java.util.concurrent.LinkedTransferQueue@25add9d
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@25add9d
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][watcher][T#2]" ID=178 WAITING on java.util.concurrent.CountDownLatch$Sync@7a110c74
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@7a110c74
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2> 	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  2> 	at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:85)
  2> 	at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:374)
  2> 	at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:365)
  2> 	at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:322)
  2> 	- locked org.elasticsearch.action.bulk.BulkProcessor@5aa2d6a2
  2> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:305)
  2> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:301)
  2> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:287)
  2> 	at org.elasticsearch.xpack.watcher.history.HistoryStore.put(HistoryStore.java:49)
  2> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:314)
  2> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:394)
  2> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$$Lambda$2855/790582040.run(Unknown Source)
  2> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:543)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@51a655bd
  2> "elasticsearch[node_s0][get][T#3]" ID=177 WAITING on java.util.concurrent.LinkedTransferQueue@b2ca6ab
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@b2ca6ab
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][warmer][T#1]" ID=176 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@5f4e2466
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@5f4e2466
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][write][T#3]" ID=175 WAITING on java.util.concurrent.LinkedTransferQueue@1315eeeb
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@1315eeeb
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][index][T#1]" ID=174 WAITING on java.util.concurrent.LinkedTransferQueue@25add9d
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@25add9d
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][get][T#2]" ID=173 WAITING on java.util.concurrent.LinkedTransferQueue@b2ca6ab
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@b2ca6ab
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][watcher][T#1]" ID=172 WAITING on java.util.concurrent.LinkedTransferQueue@61152251
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@61152251
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][write][T#2]" ID=171 WAITING on java.util.concurrent.LinkedTransferQueue@1315eeeb
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@1315eeeb
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][get][T#1]" ID=170 WAITING on java.util.concurrent.LinkedTransferQueue@b2ca6ab
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@b2ca6ab
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][search][T#5]" ID=169 WAITING on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][search][T#4]" ID=168 WAITING on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][search][T#3]" ID=167 WAITING on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][search][T#2]" ID=166 WAITING on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][search][T#1]" ID=165 WAITING on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@f62d926
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][refresh][T#1]" ID=164 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@21a1b1b8
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@21a1b1b8
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][write][T#1]" ID=163 WAITING on java.util.concurrent.LinkedTransferQueue@1315eeeb
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.LinkedTransferQueue@1315eeeb
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][generic][T#4]" ID=160 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@1dff61d1
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@1dff61d1
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][generic][T#3]" ID=159 WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@1dff61d1
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@1dff61d1
  2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][management][T#1]" ID=158 TIMED_WAITING on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6b0359e4
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue@6b0359e4
  2> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2> 	at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
  2> 	at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2> 	at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1277)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][generic][T#2]" ID=157 BLOCKED on org.elasticsearch.action.bulk.BulkProcessor@5aa2d6a2 owned by "elasticsearch[node_s0][watcher][T#2]" ID=178
  2> 	at org.elasticsearch.action.bulk.BulkProcessor$Flush.run(BulkProcessor.java:402)
  2> 	- blocked on org.elasticsearch.action.bulk.BulkProcessor@5aa2d6a2
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660)
  2> 	at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:182)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759)
  2> 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@14c21f8f
  2> "elasticsearch[node_s0][generic][T#1]" ID=154 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55b0cf95
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55b0cf95
  2> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
  2> 	at org.elasticsearch.xpack.watcher.execution.CurrentExecutions.sealAndAwaitEmpty(CurrentExecutions.java:79)
  2> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$clearExecutions$7(ExecutionService.java:525)
  2> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$$Lambda$2420/1555822424.run(Unknown Source)
  2> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@48dd4517
  2> "Connection evictor" ID=149 TIMED_WAITING
  2> 	at java.lang.Thread.sleep(Native Method)
  2> 	at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][scheduler][T#1]" ID=148 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4c8287ec
  2> 	at sun.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4c8287ec
  2> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  2> 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "elasticsearch[node_s0][[timer]]" ID=147 TIMED_WAITING
  2> 	at java.lang.Thread.sleep(Native Method)
  2> 	at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:548)
  2> "SUITE-SingleNodeTests-seed#[86F308EEE121B18C]-worker" ID=146 BLOCKED on org.elasticsearch.action.bulk.BulkProcessor@5aa2d6a2 owned by "elasticsearch[node_s0][watcher][T#2]" ID=178
  2> 	at org.elasticsearch.action.bulk.BulkProcessor.flush(BulkProcessor.java:391)
  2> 	- blocked on org.elasticsearch.action.bulk.BulkProcessor@5aa2d6a2
  2> 	at org.elasticsearch.xpack.watcher.Watcher.close(Watcher.java:700)
  2> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103)
  2> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85)
  2> 	at org.elasticsearch.xpack.core.LocalStateCompositeXPackPlugin.close(LocalStateCompositeXPackPlugin.java:434)
  2> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103)
  2> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85)
  2> 	at org.elasticsearch.node.Node.close(Node.java:928)
  2> 	- locked org.elasticsearch.node.MockNode@6082c3b5
  2> 	at org.elasticsearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1005)
  2> 	at org.elasticsearch.core.internal.io.IOUtils.closeWhileHandlingException(IOUtils.java:145)
  2> 	at org.elasticsearch.test.InternalTestCluster.close(InternalTestCluster.java:836)
  2> 	- locked org.elasticsearch.test.InternalTestCluster@695c79d
  2> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103)
  2> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85)
  2> 	at org.elasticsearch.test.ESIntegTestCase.clearClusters(ESIntegTestCase.java:540)
  2> 	at org.elasticsearch.test.ESIntegTestCase.afterClass(ESIntegTestCase.java:2215)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.lang.reflect.Method.invoke(Method.java:498)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901)
  2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2> 	at java.lang.Thread.run(Thread.java:748)
  2> "SUITE-SingleNodeTests-seed#[86F308EEE121B18C]" ID=145 RUNNABLE
  2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
  2> 	- locked java.lang.Object@558a6cbd
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:708)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629)
  2> "JUnit4-serializer-daemon" ID=18 TIMED_WAITING
  2> 	at java.lang.Thread.sleep(Native Method)
  2> 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@def2c07
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@def2c07
  2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
  2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
  2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
  2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@7b86d964
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.Reference$Lock@7b86d964
  2> 	at java.lang.Object.wait(Object.java:502)
  2> 	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
  2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
  2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@2a3e53df
  2> 	at java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@2a3e53df
  2> 	at java.lang.Thread.join(Thread.java:1252)
  2> 	at java.lang.Thread.join(Thread.java:1326)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:639)
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:496)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394)
  2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
  2> ^^==============================================
  2> dec. 18, 2018 8:44:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 27 leaked thread(s).
  2> dec. 18, 2018 8:44:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 27 threads leaked from SUITE scope at org.elasticsearch.xpack.watcher.test.integration.SingleNodeTests: 
  2>    1) Thread[id=146, name=SUITE-SingleNodeTests-seed#[86F308EEE121B18C]-worker, state=BLOCKED, group=TGRP-SingleNodeTests]
  2>         at org.elasticsearch.action.bulk.BulkProcessor.flush(BulkProcessor.java:391)
  2>         at org.elasticsearch.xpack.watcher.Watcher.close(Watcher.java:700)
  2>         at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103)
  2>         at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85)
  2>         at org.elasticsearch.xpack.core.LocalStateCompositeXPackPlugin.close(LocalStateCompositeXPackPlugin.java:434)
  2>         at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103)
  2>         at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85)
  2>         at org.elasticsearch.node.Node.close(Node.java:928)
  2>         at org.elasticsearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1005)
  2>         at org.elasticsearch.core.internal.io.IOUtils.closeWhileHandlingException(IOUtils.java:145)
  2>         at org.elasticsearch.test.InternalTestCluster.close(InternalTestCluster.java:836)
  2>         at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103)
  2>         at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85)
  2>         at org.elasticsearch.test.ESIntegTestCase.clearClusters(ESIntegTestCase.java:540)
  2>         at org.elasticsearch.test.ESIntegTestCase.afterClass(ESIntegTestCase.java:2215)
  2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.lang.reflect.Method.invoke(Method.java:498)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901)
  2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    2) Thread[id=148, name=elasticsearch[node_s0][scheduler][T#1], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    3) Thread[id=174, name=elasticsearch[node_s0][index][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    4) Thread[id=169, name=elasticsearch[node_s0][search][T#5], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    5) Thread[id=175, name=elasticsearch[node_s0][write][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    6) Thread[id=171, name=elasticsearch[node_s0][write][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    7) Thread[id=177, name=elasticsearch[node_s0][get][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    8) Thread[id=159, name=elasticsearch[node_s0][generic][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>    9) Thread[id=149, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>         at java.lang.Thread.sleep(Native Method)
  2>         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   10) Thread[id=164, name=elasticsearch[node_s0][refresh][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   11) Thread[id=170, name=elasticsearch[node_s0][get][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   12) Thread[id=180, name=elasticsearch[node_s0][management][T#2], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1277)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   13) Thread[id=165, name=elasticsearch[node_s0][search][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   14) Thread[id=178, name=elasticsearch[node_s0][watcher][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  2>         at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  2>         at org.elasticsearch.action.bulk.BulkRequestHandler.execute(BulkRequestHandler.java:85)
  2>         at org.elasticsearch.action.bulk.BulkProcessor.execute(BulkProcessor.java:374)
  2>         at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:365)
  2>         at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:322)
  2>         at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:305)
  2>         at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:301)
  2>         at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:287)
  2>         at org.elasticsearch.xpack.watcher.history.HistoryStore.put(HistoryStore.java:49)
  2>         at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:314)
  2>         at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:394)
  2>         at org.elasticsearch.xpack.watcher.execution.ExecutionService$$Lambda$2855/790582040.run(Unknown Source)
  2>         at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:543)
  2>         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   15) Thread[id=166, name=elasticsearch[node_s0][search][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   16) Thread[id=147, name=elasticsearch[node_s0][[timer]], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>         at java.lang.Thread.sleep(Native Method)
  2>         at org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:548)
  2>   17) Thread[id=173, name=elasticsearch[node_s0][get][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   18) Thread[id=168, name=elasticsearch[node_s0][search][T#4], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   19) Thread[id=160, name=elasticsearch[node_s0][generic][T#4], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   20) Thread[id=158, name=elasticsearch[node_s0][management][T#1], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1277)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   21) Thread[id=176, name=elasticsearch[node_s0][warmer][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   22) Thread[id=163, name=elasticsearch[node_s0][write][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   23) Thread[id=172, name=elasticsearch[node_s0][watcher][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   24) Thread[id=157, name=elasticsearch[node_s0][generic][T#2], state=BLOCKED, group=TGRP-SingleNodeTests]
  2>         at org.elasticsearch.action.bulk.BulkProcessor$Flush.run(BulkProcessor.java:402)
  2>         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660)
  2>         at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:182)
  2>         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759)
  2>         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   25) Thread[id=179, name=elasticsearch[node_s0][index][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   26) Thread[id=167, name=elasticsearch[node_s0][search][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  2>         at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
  2>         at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
  2>         at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269)
  2>         at org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
  2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2>   27) Thread[id=154, name=elasticsearch[node_s0][generic][T#1], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>         at sun.misc.Unsafe.park(Native Method)
  2>         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
  2>         at org.elasticsearch.xpack.watcher.execution.CurrentExecutions.sealAndAwaitEmpty(CurrentExecutions.java:79)
  2>         at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$clearExecutions$7(ExecutionService.java:525)
  2>         at org.elasticsearch.xpack.watcher.execution.ExecutionService$$Lambda$2420/1555822424.run(Unknown Source)
  2>         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:660)
  2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  2>         at java.lang.Thread.run(Thread.java:748)
  2> dec. 18, 2018 8:44:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=146, name=SUITE-SingleNodeTests-seed#[86F308EEE121B18C]-worker, state=BLOCKED, group=TGRP-SingleNodeTests]
  2>    2) Thread[id=148, name=elasticsearch[node_s0][scheduler][T#1], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>    3) Thread[id=174, name=elasticsearch[node_s0][index][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>    4) Thread[id=169, name=elasticsearch[node_s0][search][T#5], state=WAITING, group=TGRP-SingleNodeTests]
  2>    5) Thread[id=175, name=elasticsearch[node_s0][write][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>    6) Thread[id=171, name=elasticsearch[node_s0][write][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>    7) Thread[id=177, name=elasticsearch[node_s0][get][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>    8) Thread[id=159, name=elasticsearch[node_s0][generic][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>    9) Thread[id=149, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>   10) Thread[id=164, name=elasticsearch[node_s0][refresh][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>   11) Thread[id=170, name=elasticsearch[node_s0][get][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>   12) Thread[id=180, name=elasticsearch[node_s0][management][T#2], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>   13) Thread[id=165, name=elasticsearch[node_s0][search][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>   14) Thread[id=178, name=elasticsearch[node_s0][watcher][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>   15) Thread[id=166, name=elasticsearch[node_s0][search][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>   16) Thread[id=147, name=elasticsearch[node_s0][[timer]], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>   17) Thread[id=173, name=elasticsearch[node_s0][get][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>   18) Thread[id=168, name=elasticsearch[node_s0][search][T#4], state=WAITING, group=TGRP-SingleNodeTests]
  2>   19) Thread[id=160, name=elasticsearch[node_s0][generic][T#4], state=WAITING, group=TGRP-SingleNodeTests]
  2>   20) Thread[id=158, name=elasticsearch[node_s0][management][T#1], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2>   21) Thread[id=176, name=elasticsearch[node_s0][warmer][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>   22) Thread[id=163, name=elasticsearch[node_s0][write][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>   23) Thread[id=172, name=elasticsearch[node_s0][watcher][T#1], state=WAITING, group=TGRP-SingleNodeTests]
  2>   24) Thread[id=157, name=elasticsearch[node_s0][generic][T#2], state=BLOCKED, group=TGRP-SingleNodeTests]
  2>   25) Thread[id=179, name=elasticsearch[node_s0][index][T#2], state=WAITING, group=TGRP-SingleNodeTests]
  2>   26) Thread[id=167, name=elasticsearch[node_s0][search][T#3], state=WAITING, group=TGRP-SingleNodeTests]
  2>   27) Thread[id=154, name=elasticsearch[node_s0][generic][T#1], state=TIMED_WAITING, group=TGRP-SingleNodeTests]
  2> Dec 18, 2018 3:14:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:watcher:test -Dtests.seed=86F308EEE121B18C -Dtests.class=org.elasticsearch.xpack.watcher.test.integration.SingleNodeTests -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Etc/UTC -Dcompiler.java=11 -Druntime.java=8
  2> NOTE: test params are: codec=Asserting(Lucene70): {result.actions.status=PostingsFormat(name=Asserting), result.input.status=PostingsFormat(name=Asserting), _type=PostingsFormat(name=Asserting), trigger_event.type=PostingsFormat(name=Asserting), result.actions.logging.logged_text=PostingsFormat(name=Asserting), node=PostingsFormat(name=Asserting), result.condition.status=PostingsFormat(name=Asserting), watch_id=PostingsFormat(name=Asserting), result.actions.id=PostingsFormat(name=Asserting), result.actions.logging.logged_text.keyword=PostingsFormat(name=Asserting), _id=PostingsFormat(name=Asserting), result.actions.type=PostingsFormat(name=Asserting), result.condition.met=PostingsFormat(name=Asserting), state=PostingsFormat(name=Asserting), result.condition.type=PostingsFormat(name=Asserting), result.input.type=PostingsFormat(name=Asserting)}, docValues:{result.execution_duration=DocValuesFormat(name=Asserting), result.actions.status=DocValuesFormat(name=Asserting), result.input.status=DocValuesFormat(name=Asserting), trigger_event.type=DocValuesFormat(name=Lucene70), _seq_no=DocValuesFormat(name=Lucene70), node=DocValuesFormat(name=Asserting), result.execution_time=DocValuesFormat(name=Lucene70), result.condition.status=DocValuesFormat(name=Lucene70), trigger_event.schedule.scheduled_time=DocValuesFormat(name=Lucene70), watch_id=DocValuesFormat(name=Lucene70), result.actions.id=DocValuesFormat(name=Lucene70), result.actions.logging.logged_text.keyword=DocValuesFormat(name=Lucene70), state=DocValuesFormat(name=Lucene70), result.condition.met=DocValuesFormat(name=Asserting), result.actions.type=DocValuesFormat(name=Asserting), _primary_term=DocValuesFormat(name=Asserting), _version=DocValuesFormat(name=Asserting), result.input.type=DocValuesFormat(name=Lucene70), result.condition.type=DocValuesFormat(name=Lucene70), trigger_event.triggered_time=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1059, maxMBSortInHeap=6.64962286101513, sim=RandomSimilarity(queryNorm=true): {}, locale=lv-LV, timezone=Asia/Colombo
  2> NOTE: Linux 4.15.0-1025-gcp amd64/Oracle Corporation 1.8.0_192 (64-bit)/cpus=16,threads=1,free=357428112,total=516947968
  2> NOTE: All tests run in this JVM: [HipChatActionTests, XContentSourceTests, SlackMessageTests, DeleteWatchTests, ScriptConditionSearchTests, SizeLimitInputStreamTests, ExecutableSlackActionTests, EncryptSensitiveDataBootstrapCheckTests, SingleNodeTests]

Looks as if there is a problem with closing the bulk processor used in the history store and in the triggered watch store when shutting down.
@spinscale spinscale added >test-failure Triaged test failures from CI :Data Management/Watcher labels Dec 18, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@tvernum
Copy link
Contributor

tvernum commented Dec 19, 2018

Also on master

This one also caused an additional test failure, but I think they're related:

org.elasticsearch.xpack.watcher.transport.action.execute.ExecuteWatchTests testExecuteAllDefaults
org.elasticsearch.xpack.watcher.test.integration.SingleNodeTests testThatLoadingWithNonExistingIndexWorks

alpar-t added a commit that referenced this issue Jan 9, 2019
@alpar-t
Mute watcher SingleNodeTests
ffa0692 
Tracking:  #36782
@alpar-t
Copy link
Contributor

alpar-t commented Jan 9, 2019

Another occurrence: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/910/console
muted test in 911095f33b0

alpar-t added a commit that referenced this issue Jan 10, 2019
@alpar-t
Mute watcher SingleNodeTests
3d66764 
Tracking:  #36782
@tlrx tlrx self-assigned this Feb 1, 2019
tlrx added a commit to tlrx/elasticsearch that referenced this issue Feb 1, 2019
@tlrx
Fix SingleNodeTests
d1e4fe8 
Closes elastic#36782
@tlrx tlrx mentioned this issue Feb 1, 2019
Closed
@tlrx tlrx removed their assignment Feb 1, 2019
@hub-cap hub-cap self-assigned this Feb 8, 2019
@jdconrad jdconrad self-assigned this Feb 8, 2019
@jdconrad jdconrad mentioned this issue Feb 11, 2019
Closed
@jdconrad
Copy link
Contributor

I've been working on reproducing this failure locally for about 2 days now by trying to update the timing using sleeps in places that appear to be related to the failure, but have been unsuccessful in getting it to recur. The shutdown process intuitively seems a bit suspect to me since it appears that watches can potentially be executed during the different stages of stopping the service if there is already a watch queued up, but I have no evidence that's it's the direct cause or technically incorrect. I would like to re-enable this test on master if there are no objections with the caveat that we add the stack trace to the warning message for "failed to execute watch" so that we have more information should it happen again.

@jdconrad
Copy link
Contributor

@hub-cap What do you think ^^

@jakelandis
Copy link
Contributor

+1 to re-enable with additional information that may help track down the root cause. These can be frustrating difficult to reproduce on command.

If this happens again we can re-mute but grab the additional info.

@jdconrad jdconrad mentioned this issue Feb 13, 2019
Merged
@talevy
Copy link
Contributor

talevy commented Feb 19, 2019

happened again!

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/2097/console

@talevy
Copy link
Contributor

talevy commented Feb 19, 2019

here are some logs including the DEBUG logs that were exposed after unmuting

  1> [2019-02-19T10:31:51,037][DEBUG][o.e.x.w.e.ExecutionService] [node_s0] failed to execute watch [KRDCIvAgjObVlaGDCszH]
  1> org.elasticsearch.index.IndexNotFoundException: no such index [.watches]
  1> 	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:176) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:115) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:263) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.doStart(TransportInstanceSingleOperationAction.java:146) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$2.onNewClusterState(TransportInstanceSingleOperationAction.java:236) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:308) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:166) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:113) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction.retry(TransportInstanceSingleOperationAction.java:233) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.handleException(TransportInstanceSingleOperationAction.java:207) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1118) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1230) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1204) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler$1.onFailure(TransportInstanceSingleOperationAction.java:270) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.update.TransportUpdateAction.handleUpdateFailureWithRetry(TransportUpdateAction.java:264) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.update.TransportUpdateAction.lambda$shardOperation$3(TransportUpdateAction.java:214) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:69) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:119) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:444) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:439) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:74) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:929) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:914) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:315) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:234) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.service.ClusterApplierService.doStop(ClusterApplierService.java:174) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:79) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.cluster.service.ClusterService.doStop(ClusterService.java:108) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:79) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.node.Node.stop(Node.java:762) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.node.Node.close(Node.java:785) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1044) ~[framework-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.closeWhileHandlingException(IOUtils.java:145) ~[elasticsearch-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.test.InternalTestCluster.close(InternalTestCluster.java:853) ~[framework-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85) ~[elasticsearch-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.test.ESIntegTestCase.clearClusters(ESIntegTestCase.java:542) ~[framework-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.test.ESIntegTestCase.afterClass(ESIntegTestCase.java:2207) ~[framework-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:901) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00]
  1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00]
  1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00]
  1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[lucene-test-framework-8.0.0-snapshot-83f9835.jar:8.0.0-snapshot-83f9835 83f9835a47a00a2ec58a4cf5fc0d492497cf7898 - jpountz - 2019-01-21 13:06:00]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
  1> [2019-02-19T10:31:51,074][ERROR][o.e.x.w.e.ExecutionService] [node_s0] failed to update watch record [KRDCIvAgjObVlaGDCszH_58ef3aad-25a4-4ec9-9268-7cc7a5cb2b33-2019-02-18T23:31:50.422Z]
  1> java.lang.IllegalStateException: bulk process already closed
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.ensureOpen(BulkProcessor.java:318) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:323) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:308) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:304) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:290) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at org.elasticsearch.xpack.watcher.history.HistoryStore.put(HistoryStore.java:52) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:319) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:400) ~[main/:?]
  1> 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:549) [main/:?]
  1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
  1> [2019-02-19T10:32:01,075][INFO ][o.e.n.Node               ] [suite] closed
ERROR   0.00s J4 | SingleNodeTests (suite) <<< FAILURES!
   > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=97, name=elasticsearch[node_s0][watcher][T#5], state=RUNNABLE, group=TGRP-SingleNodeTests]
   > Caused by: java.lang.IllegalStateException: bulk process already closed
   > 	at __randomizedtesting.SeedInfo.seed([A480EFF62EBB4657]:0)
   > 	at org.elasticsearch.action.bulk.BulkProcessor.ensureOpen(BulkProcessor.java:318)
   > 	at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:323)
   > 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:308)
   > 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:304)
   > 	at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:297)
   > 	at org.elasticsearch.xpack.watcher.execution.TriggeredWatchStore.delete(TriggeredWatchStore.java:119)
   > 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:327)
   > 	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:400)
   > 	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:549)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   > 	at java.lang.Thread.run(Thread.java:748)
Completed [95/140] on J4 in 17.50s, 1 test, 1 error <<< FAILURES!

@jdconrad jdconrad mentioned this issue Feb 19, 2019
Merged
jdconrad added a commit that referenced this issue Feb 19, 2019
@jdconrad
Mute SingleNodeTests (#39156)
92f7b9e 
Relates to #36782
@jdconrad
Copy link
Contributor

@talevy thanks, will continue to investigate

@jdconrad
Copy link
Contributor

I think the error that happened above is actually slightly different from the original, but I believe they all stem from the same thing. It appears to me that watcher initiates it's shutdown and drains the queue, but it's possible that something has already made it through the queue already on a different thread after the shutdown takes place and the watcher indices are already gone. Watcher then tries to recreate the indices, but it cannot because of the ordering of shutdown in Node. I believe the solution is to ensure that no watches are already in progress during a watcher shutdown.

@jakelandis Does this make sense? Or am I off base here?

@jakelandis jakelandis mentioned this issue May 22, 2019
Closed
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Feb 24, 2020
@martijnvg
unmute watcher single node test case
a0c9889 
relates to elastic#36782
@martijnvg martijnvg assigned martijnvg and unassigned hub-cap and jdconrad Feb 24, 2020
@martijnvg martijnvg mentioned this issue Feb 24, 2020
Merged
martijnvg added a commit that referenced this issue Feb 28, 2020
@martijnvg
unmute watcher single node test case (#52693)
d03ac93 
relates to #36782
martijnvg added a commit that referenced this issue Mar 4, 2020
@martijnvg
unmute watcher single node test case
b77f674 
relates to #36782
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Mar 4, 2020
@martijnvg
Mute testMonitorClusterHealth test
d566649 
Relates to elastic#36782
@martijnvg martijnvg mentioned this issue Mar 4, 2020
Merged
martijnvg added a commit that referenced this issue Mar 4, 2020
@martijnvg
Mute testMonitorClusterHealth test (#53109)
9918a8f 
Relates to #36782
martijnvg added a commit that referenced this issue Mar 4, 2020
@martijnvg
Mute testMonitorClusterHealth test (#53109)
2e325e2 
Relates to #36782
@martijnvg
Copy link
Member

This test has been enabled a for a while now and hasn't yet failed.
Closing this issue, if this test does fail again then a new issue should be opened.
(so that there is a clean history)

@tlrx tlrx mentioned this issue Mar 24, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@martijnvg martijnvg

Labels
:Data Management/Watcher >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@talevy @martijnvg @hub-cap @tlrx @spinscale @jakelandis @jdconrad @tvernum @alpar-t @elasticmachine