live destroy hangs waiting for KCC resource

[bharathkkb]

Expected behavior

destroy returns successful

Actual behavior

destroy seems to hang although resource is no longer present in cluster (and in GCP)

Information

kpt version
1.0.0-beta.12

Steps to reproduce the behavior

• Create a CC cluster
• Follow https://github.com/GoogleCloudPlatform/blueprints/tree/main/catalog/bucket to deploy bucket blueprint

 kpt live apply
storagebucket.storage.cnrm.cloud.google.com/test-project-bucket-1-bucket2 created
1 resource(s) applied. 1 created, 0 unchanged, 0 configured, 0 failed
storagebucket.storage.cnrm.cloud.google.com/test-project-bucket-1-bucket2 reconcile pending
storagebucket.storage.cnrm.cloud.google.com/test-project-bucket-1-bucket2 reconciled

• Verify bucket exists

k -n config-control get StorageBucket
NAME                          AGE   READY   STATUS     STATUS AGE
test-project-bucket-1-bucket2   72s   True    UpToDate   70s

• Destroy resource

kpt live destroy
storagebucket.storage.cnrm.cloud.google.com/test-project-bucket-1-bucket2 deleted
1 resource(s) deleted, 0 skipped, 0 failed to delete
storagebucket.storage.cnrm.cloud.google.com/test-project-bucket-1-bucket2 reconcile pending

At this point kpt seems to hang. I checked via kubectl to confirm it was deleted

 k -n config-control get StorageBucket
No resources found in config-control namespace.

When I exited and retried again it seemed to exit with 0

kpt live destroy
echo $?
0

[droot]

@mortent This is potential blocker for promoting beta.12 in gcloud. so pl. take a look.

[mortent]

@rquitales Is this something you can take a look at?

[rquitales]

@rquitales Is this something you can take a look at?

Will look into this.

[rquitales]

This bug occurs in our custom StatusReader for Config Connector (CC) resources. When a CC resource is deleted, the status reader returns an unknown error instead of a NotFound message, resulting in an endless reconcile loop.

The specific error handling logic branch that causes this is:
https://github.com/GoogleContainerTools/kpt/blob/ef1a189ec04c83de6299ab9d252805d963f32ec8/pkg/status/configconnector.go#L57-L62

I'll send out a patch for this soon.