Clean namespace handover

[yycptt]

What changed?

• Block creation/update operation if namespace is in handover state
• Use immediate task max read level as max replication taskID for checking is handover is done and ready for handover. The previously value from replication ack manager may miss some taskIDs as the value is updated through task notification which is async.

Why?

• Make sure a clean handover for namespace migration and two cluster won't have conflicting history.

How did you test it?

• Need to test on test cluster

Potential risks

• handover may take longer to complete & more retires with in the system during namespace failover as history service will now fail workflow creation/update requests.

Is hotfix candidate?

• no

[meiliang86]

Potential risks: larger unavailable window?

[meiliang86]

Is there a way to check/verify that conflict resolution never happened after graceful failover?

[yycptt]

Potential risks: larger unavailable window?

yes if shard is unstable during handover, otherwise the increase should be very small.

Is there a way to check/verify that conflict resolution never happened after graceful failover?

We can check if there's any persistence_request metric with ConflictResolveWorkflowExecution operation tag.