[CI] ActivateWatchTests testDeactivateAndActivate failing

[dimitris-athanasiou]

Build scan:
https://gradle-enterprise.elastic.co/s/7tp3knsxy7zok/tests/:x-pack:plugin:watcher:internalClusterTest/org.elasticsearch.xpack.watcher.transport.action.activate.ActivateWatchTests/testDeactivateAndActivate

Reproduction line:
./gradlew ':x-pack:plugin:watcher:internalClusterTest' --tests "org.elasticsearch.xpack.watcher.transport.action.activate.ActivateWatchTests.testDeactivateAndActivate" -Dtests.seed=1A848E0804358E18 -Dtests.locale=zh-Hans-SG -Dtests.timezone=America/Adak -Druntime.java=17

Applicable branches:
8.0

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.watcher.transport.action.activate.ActivateWatchTests&tests.test=testDeactivateAndActivate

Failure excerpt:

java.lang.IllegalArgumentException: index [.ds-.watcher-history-14-2021.12.13-000001] is the write index for data stream [.watcher-history-14] and cannot be deleted

  at __randomizedtesting.SeedInfo.seed([1A848E0804358E18:BADCF6F5CB6C5864]:0)
  at org.elasticsearch.cluster.metadata.MetadataDeleteIndexService.deleteIndices(MetadataDeleteIndexService.java:90)
  at org.elasticsearch.cluster.metadata.MetadataDeleteIndexService$1.execute(MetadataDeleteIndexService.java:68)
  at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:51)
  at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:828)
  at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:395)
  at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:235)
  at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:167)
  at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:146)
  at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:202)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:716)
  at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:262)
  at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:225)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

Closing as a duplicate of #66495, which has more discussion.