Make nodepool concurrent ops scale better

[Xylosma]

The purpose of this change is to stagger operation creations, but still keeps operations concurrent, so that operations are not bottlenecked on creation and fail.
To achieve the above,

• a cluster-scope lock is held for operation creation
• cluster-scope lock is released after creation is done
• cluster-scope read lock is held while waiting for operation to complete (as before)

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

container: make nodepool concurrent operations scale better

[github-actions]

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

@ScottSuarez, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 124 insertions(+), 11 deletions(-))
google-beta provider: Diff ( 2 files changed, 124 insertions(+), 11 deletions(-))

[modular-magician]

Tests analytics

Total tests: 219
Passed tests: 207
Skipped tests: 12
Affected tests: 0

Click here to see the affected service packages

• container

🟢 All tests passed!

View the build log

[ScottSuarez]

We should look to upstream this locking behavior to retryWhileIncompatibleOperation.

This resource already has custom locking behavior on these calls. If we want to customize that behavior we should customize that function. That would mean passing that operation back and doing the wait within that function rather then introducing a secondary locking mechanism. Otherwise this will be difficult to interpret and new additions will also need to remember to implement this same behavior you have here.

https://github.com/hashicorp/terraform-provider-google-beta/blob/d02bfdd327ae6fbef0aa744c8c4c51c3abd3ba11/google-beta/services/container/resource_container_node_pool.go#L1693

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 144 insertions(+), 174 deletions(-))
google-beta provider: Diff ( 2 files changed, 144 insertions(+), 174 deletions(-))

[modular-magician]

Tests analytics

Total tests: 219
Passed tests: 207
Skipped tests: 12
Affected tests: 0

Click here to see the affected service packages

• container

🟢 All tests passed!

View the build log

[ScottSuarez]

I think it would be a good idea to sync over a gvc and talk about this a bit!

[alexmorozov]

Looks good overall, thanks for making the change. I'd ask to uplevel the comments in the code that refer to which functions take which locks. That part is more or less evident from the code itself - but it's not clear to an external viewer why that is done. You explain that in a later comment, but I would not expect the reader to be able to find it without guidance.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 147 insertions(+), 175 deletions(-))
google-beta provider: Diff ( 2 files changed, 147 insertions(+), 175 deletions(-))

[ScottSuarez]

Thanks for your patience here :)

[Xylosma]

Thank you Scott and Alex for the review!