RG created with 10 PVs. Verified data writes on PowerScale SSH sessions. Tested all replication actions. Simulated QuotaScan error and sync action resolved the failed policy. [INFO] found no active sync jobs, starting a new one [DEBUG] -------------------------- GOISILON HTTP REQUEST ------------------------- POST /platform/11/sync/jobs/ HTTP/1.1 Host: :8080 Authorization: root:****** Content-Type: application/json {"id":"sl-csi-ns51--Five_Minutes"} [DEBUG] -------------------------- GOISILON HTTP RESPONSE ------------------------- HTTP/1.1 500 Internal Server Error [DEBUG] -------------------------- GOISILON HTTP REQUEST ------------------------- GET /platform/11/sync/reports/?policy_name=sl-csi-ns51--Five_Minutes&sort=end_time&reports_per_policy=1 HTTP/1.1 Host: :8080 Authorization: root:****** [DEBUG] -------------------------- GOISILON HTTP RESPONSE ------------------------- HTTP/1.1 200 Ok [INFO] Sync job failed with error: A new quota domain that has not finished QuotaScan has been found. Please wait for QuotaScan to finish before restarting the sync job. Quota path: /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/a: Invalid argument (unrunnable). 1 of 20 - retrying in 15s... [DEBUG] -------------------------- GOISILON HTTP REQUEST ------------------------- GET /platform/11/sync/policies/sl-csi-ns51--Five_Minutes HTTP/1.1 Host: :8080 Authorization: root:****** [DEBUG] -------------------------- GOISILON HTTP RESPONSE ------------------------- HTTP/1.1 200 Ok [DEBUG] -------------------------- GOISILON HTTP REQUEST ------------------------- PUT /platform/11/sync/policies/0bd17c1d952e3cc6b8856ea4f50c6164 HTTP/1.1 Host: :8080 Authorization: root:****** Content-Type: application/json {"enabled":true,"conflicted":false,"schedule":"when-source-modified"} [DEBUG] -------------------------- GOISILON HTTP RESPONSE ------------------------- HTTP/1.1 204 No Content [DEBUG] -------------------------- GOISILON HTTP REQUEST ------------------------- POST /platform/11/sync/jobs/ HTTP/1.1 Host: :8080 Authorization: root:****** Content-Type: application/json {"id":"sl-csi-ns51--Five_Minutes"} [DEBUG] -------------------------- GOISILON HTTP RESPONSE ------------------------- HTTP/1.1 201 Created [DEBUG] -------------------------- GOISILON HTTP REQUEST ------------------------- GET /platform/11/sync/jobs/sl-csi-ns51--Five_Minutes HTTP/1.1 Host: :8080 Authorization: root:****** [DEBUG] -------------------------- GOISILON HTTP RESPONSE ------------------------- HTTP/1.1 200 Ok Source RG: status: conditions: - condition: Replication Link State:IsSource changed from (true) to (false) time: "2023-04-10T20:26:57Z" - condition: Action FAILOVER_REMOTE succeeded time: "2023-04-10T20:25:01Z" - condition: Action ACTION_FAILBACK_DISCARD_CHANGES_LOCAL succeeded time: "2023-04-10T20:21:35Z" - condition: Action SYNC succeeded time: "2023-04-10T20:19:31Z" - condition: Action FAILBACK_LOCAL succeeded time: "2023-04-10T20:16:52Z" - condition: Action FAILOVER_REMOTE succeeded time: "2023-04-10T20:08:22Z" - condition: Action SYNC succeeded time: "2023-04-10T18:22:51Z" - condition: Action SYNC succeeded time: "2023-04-10T18:19:47Z" - condition: Action SYNC succeeded time: "2023-04-10T18:17:23Z" - condition: Action SYNC succeeded time: "2023-04-10T18:14:27Z" - condition: Action SYNC succeeded time: "2023-04-10T18:00:27Z" - condition: Action SYNC succeeded time: "2023-04-10T17:49:47Z" - condition: Action RESUME succeeded time: "2023-04-10T15:58:41Z" - condition: Action SUSPEND succeeded time: "2023-04-10T15:56:35Z" - condition: Action SYNC succeeded time: "2023-04-10T15:55:34Z" - condition: Replication Link State:IsSource changed from (false) to (true) time: "2023-04-10T15:46:57Z" lastAction: condition: Action FAILOVER_REMOTE succeeded time: "2023-04-10T20:25:01Z" replicationLinkState: isSource: false lastSuccessfulUpdate: "2023-04-10T20:26:57Z" state: SYNCHRONIZED state: Ready Target RG: status: conditions: - condition: Replication Link State:IsSource changed from (false) to (true) time: "2023-04-10T20:26:09Z" - condition: Action REPROTECT_LOCAL succeeded time: "2023-04-10T20:26:09Z" - condition: Action UNPLANNED_FAILOVER_LOCAL succeeded time: "2023-04-10T20:20:12Z" lastAction: condition: Action REPROTECT_LOCAL succeeded time: "2023-04-10T20:26:09Z" replicationLinkState: isSource: true lastSuccessfulUpdate: "2023-04-10T20:26:09Z" state: SYNCHRONIZED state: Ready Switched to original source. Created 11th and then 12th PV for the same RG. Simulated QuotaScan error and CreateRemoteVolume() call resolved the failed policy. Source PowerScale cluster: SRC-1# pwd /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes SRC-1# SRC-1# ls sl-k8s-138054ff17 sl-k8s-5fe7159325 sl-k8s-6f214a1ac6 sl-k8s-9ba5f795f2 sl-k8s-dbf22e7e3b sl-k8s-ee4744f153 sl-k8s-57c14f47ed sl-k8s-6b84d3310e sl-k8s-740c79006e sl-k8s-bd081c0893 sl-k8s-ee11cd9a2a sl-k8s-f2d8e9c009 SRC-1# SRC-1# isi quota quotas list | grep ns51 | wc -l 12 SRC-1# SRC-1# isi quota quotas list | grep ns51 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-9ba5f795f2 No 9.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-5fe7159325 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-57c14f47ed No 10.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee11cd9a2a No 7.00G - - 1.42M 1.00 : 1 0.71 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6b84d3310e No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6f214a1ac6 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-740c79006e No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-dbf22e7e3b No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-bd081c0893 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee4744f153 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-138054ff17 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-f2d8e9c009 No 7.00G - - 0.00 - 0.00 : 1 SRC-1# SRC-1# SRC-1# isi nfs exports list | grep ns51 | wc -l 12 SRC-1# SRC-1# isi nfs exports list | grep ns51 11495 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-5fe7159325 CSI_QUOTA_ID:4RV-AwEAAAAAAAAAAAAAQJYrAAAAAAAA 11496 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-740c79006e CSI_QUOTA_ID:7kTQAwEAAAAAAAAAAAAAQJUrAAAAAAAA 11497 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee11cd9a2a CSI_QUOTA_ID:bxi8AwEAAAAAAAAAAAAAQJcrAAAAAAAA 11498 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-dbf22e7e3b CSI_QUOTA_ID:70TQAwEAAAAAAAAAAAAAQJgrAAAAAAAA 11499 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-bd081c0893 CSI_QUOTA_ID:8ETQAwEAAAAAAAAAAAAAQJkrAAAAAAAA 11500 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee4744f153 CSI_QUOTA_ID:8UTQAwEAAAAAAAAAAAAAQJorAAAAAAAA 11501 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-138054ff17 CSI_QUOTA_ID:8kTQAwEAAAAAAAAAAAAAQJsrAAAAAAAA 11502 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-f2d8e9c009 CSI_QUOTA_ID:80TQAwEAAAAAAAAAAAAAQJwrAAAAAAAA 11503 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6b84d3310e CSI_QUOTA_ID:cBi8AwEAAAAAAAAAAAAAQJ0rAAAAAAAA 11504 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6f214a1ac6 CSI_QUOTA_ID:cRi8AwEAAAAAAAAAAAAAQJ4rAAAAAAAA 11523 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-9ba5f795f2 CSI_QUOTA_ID:SwtpAwEAAAAAAAAAAAAAQK4rAAAAAAAA 11524 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-57c14f47ed CSI_QUOTA_ID:61OZAwEAAAAAAAAAAAAAQK8rAAAAAAAA SRC-1# SRC-1# ls -ltr total 56 drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-5fe7159325 drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-740c79006e drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-dbf22e7e3b drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-bd081c0893 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-ee4744f153 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-138054ff17 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-f2d8e9c009 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-6b84d3310e drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-6f214a1ac6 drwxrwxrwx 2 root wheel 259 Apr 10 20:31 sl-k8s-ee11cd9a2a drwxrwxrwx 2 root wheel 0 Apr 10 21:53 sl-k8s-9ba5f795f2 drwxrwxrwx 2 root wheel 0 Apr 11 14:58 sl-k8s-57c14f47ed SRC-1# Target PowerScale cluster: TGT-1# pwd /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes TGT-1# TGT-1# ls sl-k8s-138054ff17 sl-k8s-5fe7159325 sl-k8s-6f214a1ac6 sl-k8s-9ba5f795f2 sl-k8s-dbf22e7e3b sl-k8s-ee4744f153 sl-k8s-57c14f47ed sl-k8s-6b84d3310e sl-k8s-740c79006e sl-k8s-bd081c0893 sl-k8s-ee11cd9a2a sl-k8s-f2d8e9c009 TGT-1# TGT-1# isi quota quotas list | grep ns51 | wc -l 12 TGT-1# TGT-1# isi quota quotas list | grep ns51 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-57c14f47ed No 10.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6b84d3310e No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6f214a1ac6 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-f2d8e9c009 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee4744f153 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-138054ff17 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-bd081c0893 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee11cd9a2a No 7.00G - - 1.42M 1.00 : 1 0.63 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-740c79006e No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-dbf22e7e3b No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-5fe7159325 No 7.00G - - 0.00 - 0.00 : 1 directory DEFAULT /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-9ba5f795f2 No 9.00G - - 0.00 - 0.00 : 1 TGT-1# TGT-1# isi nfs exports list | grep ns51 | wc -l 12 TGT-1# TGT-1# isi nfs exports list | grep ns51 111839 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-5fe7159325 CSI_QUOTA_ID:NAQxCgEAAAAAAAAAAAAAQBavAQAAAAAA 111840 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-740c79006e CSI_QUOTA_ID:MgQxCgEAAAAAAAAAAAAAQBevAQAAAAAA 111841 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee11cd9a2a CSI_QUOTA_ID:MQQxCgEAAAAAAAAAAAAAQBivAQAAAAAA 111842 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-dbf22e7e3b CSI_QUOTA_ID:MwQxCgEAAAAAAAAAAAAAQBmvAQAAAAAA 111843 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-ee4744f153 CSI_QUOTA_ID:FQQiCgEAAAAAAAAAAAAAQBqvAQAAAAAA 111844 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-bd081c0893 CSI_QUOTA_ID:FwQiCgEAAAAAAAAAAAAAQBuvAQAAAAAA 111845 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-f2d8e9c009 CSI_QUOTA_ID:R1dcCQEAAAAAAAAAAAAAQByvAQAAAAAA 111846 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-138054ff17 CSI_QUOTA_ID:FgQiCgEAAAAAAAAAAAAAQB2vAQAAAAAA 111847 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6f214a1ac6 CSI_QUOTA_ID:RldcCQEAAAAAAAAAAAAAQB6vAQAAAAAA 111848 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-6b84d3310e CSI_QUOTA_ID:RVdcCQEAAAAAAAAAAAAAQB-vAQAAAAAA 111899 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-9ba5f795f2 CSI_QUOTA_ID:NwVACgEAAAAAAAAAAAAAQF6vAQAAAAAA 111963 System /ifs/data/csi/1sl/sl-csi-ns51--Five_Minutes/sl-k8s-57c14f47ed CSI_QUOTA_ID:sldUCQEAAAAAAAAAAAAAQKmvAQAAAAAA TGT-1# TGT-1# ls -ltr total 56 drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-5fe7159325 drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-740c79006e drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-dbf22e7e3b drwxrwxrwx 2 root wheel 0 Apr 10 15:06 sl-k8s-bd081c0893 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-ee4744f153 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-138054ff17 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-f2d8e9c009 drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-6b84d3310e drwxrwxrwx 2 root wheel 0 Apr 10 15:07 sl-k8s-6f214a1ac6 drwxrwxrwx 2 root wheel 259 Apr 10 20:31 sl-k8s-ee11cd9a2a drwxrwxrwx 2 root wheel 0 Apr 10 21:53 sl-k8s-9ba5f795f2 drwxrwxrwx 2 root wheel 0 Apr 11 14:58 sl-k8s-57c14f47ed TGT-1#