RFC: Improve saved object migrations (#66056)

* RFC: Improve saved object migrations

* Added rolling upgrades as alternative

* Clarify advantage of re-using index for avoiding max shards open errors

* Add more detail about how dry run migrations and invalid document exports will be exposed to users

* Add separate section to highlight assumptions and tradeoffs

* Apply nits from code review

Co-authored-by: Josh Dover <[email protected]>

* Move 'Single node migrations coordinated through a lease/lock' under alternatives

* Fix 5.2.2 document lock algorithm step 3.2

* Minor clarificiations

* Minimizing data loss with mixed Kibana versions: move 7.x algorithm into alternatives and clarify 8.x algorithm

* Adds impact idempotent restrictions have on upcoming/requested migration features

* nit: clarify 4.3.1.2.3

Clarify that if documents already exist, they aren't overwritten.

Co-authored-by: Brandon Kobel <[email protected]>

* Update 4.5.2: we need only one version-specific alias, not separate read/write aliases

* Add 4.0.4 the assumption that a saved object type will only ever be registered by one plugin

* Updates 4.3.1.2: Adds mappings compatibility check, adds soft deletes to prevent lost deletes, add note about data loss with mixed plugins on the same kibana version

* Don't try to minimize data loss for unsupported configurations, update index mappings during 8.x migration

 - Move "4.5.2 Minimizing data loss with mixed Kibana versions (9.0)" to
 alternatives section. Expand this section with a more detailed algorithm as
 well as enumerating some data loss scenarios that could still occur.
 - Add the supported upgrade procedure which guarantees no undetected data loss.
 - Update "4.5.1 Migration algorithm (8.0)"
   - Added a step to update index mappings
   - If there's update version mismatch errors, complete all batches before
     restarting the migration to improve performance.

* Flesh out upgrade and rollback procedure with gracefull shutdown

* New migration algorithm, upgrade procedure based on index cloning and new ES features

* Minor: clarify that index cloning algorithm solves (2.6) but (2.7) is out of scope

* Move 'Tag objects as invalid if their transformation fails' to the alternatives section

* Don't migrate when there's orphan documents so that rollback is always possible. Updates: 4.2.1.2 Migration algorithm: Cloned index per version & 4.2.1.4 Handling documents that belong to a disabled plugin

* Link to new Elasticsearch features required for the implementation

* 4.2.1.2 Clarify the purpose of the target index postfix

* Add open questions around mappings growing over time. Fix rollback index example

* minor: fix formatting

* Update assumptions and tradeoffs to match latest algorithm

* Review feedback
 - Remove 4.0.2 since the latest algorithm resolves the tradeoff
 - 4.2.1.2 should handle v6 `.kibana` indexes

* More alternatives for 4.2.1.4 Handling documents that belong to a disabled plugin. Don't introduce breaking changes in 7.x

* Handle new ES cluster without any indices. Fix clone API parameters

* Update 4.2.1.2: transform outdated documents on every startup, only fail for unknown types in 8.x

Co-authored-by: Josh Dover <[email protected]>
Co-authored-by: Brandon Kobel <[email protected]>