[Monitoring] Issues with our api integration tests

[chrisronline]

There are a couple issues plaguing us recently with api integration test failures.

Removal of mappings.json

We removed our mappings.json from our archives which led to quite a bit of pain. You can read more about that here: #30152. We actually missed updating one of our archives, needing a fix (#31315)

The symptom of this problem comes with one of these two tests fail: cluster overview and cluster list. They fail because the bug from above leaves indices in the functional test server, which affects how aggregations are returned in subsequent tests. (I'll add more to this later)

An object mutated

This other issue was recently discovered where the standalone cluster definition was being mutated directly (instead of copied and changes only affecting the copy) meaning that the first time it was accessed, it worked as expected, but subsequent tests that read from it would not have the same object - they'd be using a mutated object.

The symptom here is that the cluster json test file had actual values for a kibana instance, when it shouldn't based on the code. Again, this happened because the multiple clusters test ran before the single cluster test. Since it ran first, it'd hit this code block which would add the standalone cluster definition object to the clusters list, which is then passed down into the kibana logic. Then, when the single cluster test ran, that object had already been mutated on the server and the test was inconsistent.

This issue honestly would have come up later for users in very unsuspecting ways, so it's good we found this now.

[elasticmachine]

Pinging @elastic/stack-monitoring