systemtest: retry deleting ILM policy

[axw]

Motivation/summary

Fix intermittent CI issues like the following, by retrying ILM policy deletion.

[2020-09-10T09:24:20.117Z] === RUN   TestTransactionAggregationShutdown
[2020-09-10T09:24:20.117Z]     elasticsearch.go:120: [400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Cannot delete policy [apm-rollover-30-days]. It is in use by one or more indices: [apm-8.0.0-error-000001]"}],"type":"illegal_argument_exception","reason":"Cannot delete policy [apm-rollover-30-days]. It is in use by one or more indices: [apm-8.0.0-error-000001]"},"status":400}

Checklist

• I have signed the Contributor License Agreement.
  - [ ] I have updated CHANGELOG.asciidoc

I have considered changes for:
- [ ] documentation
- [ ] logging (add log lines, choose appropriate log selector, etc.)
- [ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)

• automated tests (add tests for the code changes, all unit tests pass locally)
  - [ ] telemetry
  - [ ] Elasticsearch Service (https://cloud.elastic.co)
  - [ ] Elastic Cloud Enterprise (https://www.elastic.co/products/ece)
  - [ ] Elastic Cloud on Kubernetes (https://www.elastic.co/elastic-cloud-kubernetes)

How to test these changes

cd systemtest && go test -v

Related issues

https://apm-ci.elastic.co/blue/organizations/jenkins/apm-server%2Fapm-server-mbp/detail/PR-4183/3/pipeline/130

[codecov-commenter]

Codecov Report

Merging #4186 into master will decrease coverage by 0.12%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4186      +/-   ##
==========================================
- Coverage   80.72%   80.60%   -0.13%     
==========================================
  Files         154      154              
  Lines        7693     7676      -17     
==========================================
- Hits         6210     6187      -23     
- Misses       1483     1489       +6     

Impacted Files	Coverage Δ
beater/middleware/log_middleware.go	84.00% <0.00%> (-8.54%)	⬇️
publish/acker.go	83.33% <0.00%> (-8.34%)	⬇️
...ack/apm-server/aggregation/txmetrics/aggregator.go	92.24% <0.00%> (-0.41%)	⬇️

[apmmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #4186 updated]

• Start Time: 2020-09-13T05:39:21.212+0000

• Duration: 48 min 29 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3540
Skipped	145
Total	3685

Steps errors

Expand to view the steps failures

• Name: Compress

  • Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

  • Duration: 0 min 0 sec

  • Start Time: 2020-09-13T05:54:52.076+0000

  • log

• Name: Compress

  • Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

  • Duration: 0 min 0 sec

  • Start Time: 2020-09-13T06:08:36.595+0000

  • log

• Name: Test Sync

  • Description: ./script/jenkins/sync.sh

  • Duration: 3 min 50 sec

  • Start Time: 2020-09-13T05:49:55.002+0000

  • log