Implement server side retry for workflow/child workflow #915
Conversation
| newRunExecutionInfo.NextEventID = nextEventID | ||
| newRunExecutionInfo.LastFirstEventID = startedEvent.GetEventId() | ||
| // Set the history from replication task on the newStateBuilder | ||
| newRunStateBuilder.SetHistoryBuilder(newHistoryBuilderFromEvents(newRunHistory.Events, b.logger)) | ||
| sourceClusterName := b.clusterMetadata.ClusterNameForFailoverVersion(startedEvent.GetVersion()) | ||
| newRunStateBuilder.UpdateReplicationStateLastEventID(sourceClusterName, startedEvent.GetVersion(), nextEventID-1) | ||
|
|
||
| b.newRunTransferTasks = append(b.newRunTransferTasks, b.scheduleDecisionTransferTask(domainID, | ||
| b.getTaskList(newRunStateBuilder), di.ScheduleID)) | ||
| if startedAttributes.GetAttempt() == 0 { |
There was a problem hiding this comment.
if attempt is !=0, then no decision is scheduled?
this can be fine if the workflow is being replicated, however, when doing a failover and the new attempt workflow is picked up by the previous standby cluster (now active), the no worker can pick this workflow up.
this workflow will be invisible until the decision task scheduled event is replicated from active cluster (now standby after the failover). however, there is no guarantee that this decision task scheduled event will be replicated.
There was a problem hiding this comment.
the first scheduled event will be created by a backoff timer, the timer should fire and because it is now become active, it would create the scheduled event locally.
There was a problem hiding this comment.
I think the stand by timer queue process will need to be updated to handle that case. will update it.
There was a problem hiding this comment.
@yiminc the standby processor only does verification, as long as the timer task is created, the failover will handle the task processing
There was a problem hiding this comment.
the failover processing logic is the same as the active processing logic
service/history/historyEngine.go
Outdated
| @@ -1392,6 +1423,9 @@ Update_History_Loop: | |||
| // the history and try the operation again. | |||
| var updateErr error | |||
| if continueAsNewBuilder != nil { | |||
| if msBuilder.GetContinueAsNew() != nil { | |||
There was a problem hiding this comment.
i think this will always be non nil after checking the continueAsNewBuilder != nil
ref: https://github.com/uber/cadence/pull/915/files#diff-01045b0895719962d1f13440e8795d1bR2416
yiminc-zz
force-pushed
the
continue_as_new
branch
from
083f300 to
fb6aae4
Compare
yiminc-zz
changed the title
yiminc-zz
changed the title
…cadence-workflow#910)" This reverts commit 492faf0. Fix continueAsNew replication issue update doc for retry policy in idl remove unnecessary nil check handle WorkflowRetryTimerTask correctly on standby side update test for cassandra tools
yiminc-zz
force-pushed
the
continue_as_new
branch
from
5ea2bc2 to
84e65fd
Compare
This PR added server side retry for workflow/child workflow.
The change was landed #885 but reverted because of regression when doing replication for ContinueAsNew event.
The regression is fixed.