support to query/operate paused subtask after re-scheduled

[csuzhangxc]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

   1. pause-task
   2. stop the DM-worker instance to trigger re-schedule the source to another DM-worker instance
   3. pause-task again

2. What did you expect to see?

   stage: paused

3. What did you see instead?

   [2020/09/23 15:27:01.081 +08:00] [ERROR] [worker.go:418] ["fail to operate subtask stage"] [stage="{\"expect\":3,\"source\":\"mysql-replica-01\",\"task\":\"test\"}"] [error="[code=40054:class=dm-worker:scope=internal:level=high], Message: sub task with name test not found"] [errorVerbose="[code=40054:class=dm-worker:scope=internal:level=high], Message: sub task with name test not found\ngithub.jparrowsec.cn/pingcap/dm/pkg/terror.(*Error).Generate\n\t/home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/pkg/terror/terror.go:263\ngithub.jparrowsec.cn/pingcap/dm/dm/worker.(*Worker).OperateSubTask\n\t/home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/worker.go:272\ngithub.jparrowsec.cn/pingcap/dm/dm/worker.(*Worker).operateSubTaskStage\n\t/home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/worker.go:458\ngithub.jparrowsec.cn/pingcap/dm/dm/worker.(*Worker).operateSubTaskStageWithoutConfig\n\t/home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/worker.go:477\ngithub.jparrowsec.cn/pingcap/dm/dm/worker.(*Worker).handleSubTaskStage\n\t/home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/worker.go:415\ngithub.jparrowsec.cn/pingcap/dm/dm/worker.(*Worker).observeSubtaskStage\n\t/home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/worker.go:374\ngithub.jparrowsec.cn/pingcap/dm/dm/worker.(*Server).startWorker.func3\n\t/home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/server.go:822\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
   

4. Versions of the cluster

   • DM version (run dmctl -V or dm-worker -V or dm-master -V):

     v2.0.0-rc.2 at least until https://github.com/pingcap/dm/commit/55fcd6db4446ef81e9081d932355b1e99a487604
     

6889-77.log