Skip to content
This repository has been archived by the owner on May 6, 2022. It is now read-only.

Handle InProgressProperties for retries properly #1903

Closed
wants to merge 4 commits into from
Closed

Handle InProgressProperties for retries properly #1903

wants to merge 4 commits into from

Conversation

nilebox
Copy link
Contributor

@nilebox nilebox commented Apr 3, 2018

Addressing the issue blocking unlocking (no pun intended) discussed in #1872 (comment)

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Apr 3, 2018
var planExternalID string
var userInfo *v1beta1.UserInfo

if instance.Status.CurrentOperation == "" {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kibbles-n-bytes after making this change, I feel like it would be better to always have a CurrentOperation set before preparing an OSB request. In other words, instead of explicitly mitigating a potential issue with retries here, we just reorder logic in controller:

  1. First record the intent in the status
  2. Then use the recorded status as a source for OSB requests.

What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making this change is hard though, given that inProgressProperties value depends on event type (Add/Update/Delete). We could probably merge this quick fix as it is, and discuss the follow-up refactoring.

@nilebox
Copy link
Contributor Author

nilebox commented Apr 3, 2018

/retest

if instance.Status.InProgressProperties == nil {
return nil, fmt.Errorf("InProgressProperties must not be nil if the CurrentOperation is set")
}
specParameters = instance.Status.InProgressProperties.InlineParameters
Copy link
Contributor Author

@nilebox nilebox Apr 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field will be empty if the operation has started prior to upgrading to the version of Service Catalog with InlineParameters field.
Could check for nil there and copy value from the spec if it's not set, but then it could break a fix: what if parameters were empty when we started operation, and were set afterwards?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question.

@nilebox nilebox changed the title [WIP] Handle InProgressProperties for retries properly Handle InProgressProperties for retries properly Apr 3, 2018
@duglin
Copy link
Contributor

duglin commented Apr 4, 2018

@nilebox can you elaborate on why you need the inlinedParameters? And, why the same info could not be retrieved from the Parameters if we skip the redacted ones?

@nilebox
Copy link
Contributor Author

nilebox commented Apr 4, 2018

@duglin I need inlineParameters to be able to read parameters from there instead of reading from the spec, which might contain updated parameters (and we haven't finished processing the previous spec version yet).

And, why the same info could not be retrieved from the Parameters if we skip the redacted ones?

How do you skip them? Ignore strings containing <redacted>? To me it's a magic reverse engineering and a bad design. Also, parameter in the spec could actually contain a "<redacted>" value.

@duglin
Copy link
Contributor

duglin commented Apr 4, 2018

I'm confused, I thought Status.ExternalProperties were for that purpose.

@nilebox
Copy link
Contributor Author

nilebox commented Apr 4, 2018

@duglin the way svc-cat currently works is:

  1. Record start of the operation and store InProgressProperties in the status
  2. Once the operation has succeeded, copy data from InProgressProperties to ExternalProperties.

The problem is that if you fail after the step 1, you have recorded InProgressProperties, but when you retry, the OSB request is still populated with parameters from the spec (which might be different from what you recorded in InProgressProperties). What's worse - once you suceeded, controller will copy stale data from InProgressProperties to ExternalProperties.
as @kibbles-n-bytes wrote in #1872 (comment), this is not that critical while we have locks, but it will become a serious issue once we make locks optional or allow skipping locks for the same user.

This PR fixes that - once you recorded InProgressProperties, controller will populate inline parameters from InProgressProperties.InlineParameters

@@ -710,6 +710,11 @@ type ServiceInstancePropertiesState struct {
// a secret, its value will be "<redacted>" in this blob.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we clarify that this is only secrets now?

@@ -1002,7 +1002,9 @@ func (c *controller) prepareBindRequest(
}
}

parameters, parametersChecksum, rawParametersWithRedaction, err := prepareInProgressPropertyParameters(
// TODO nilebox: If the current operation is already set, we need to ignore the spec and read parameters from the status
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you going to handle this TODO here?

@@ -35,18 +35,19 @@ import (
// The second return value is a map of parameters with secret values redacted,
// replaced with "<redacted>".
// The third return value is any error that caused the function to fail.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you update the godoc here

@@ -154,34 +156,47 @@ func generateChecksumOfParameters(params map[string]interface{}) (string, error)
// 2 - a checksum for the map of parameters. This checksum is used to determine if parameters have changed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update godoc

if instance.Status.InProgressProperties == nil {
return nil, fmt.Errorf("InProgressProperties must not be nil if the CurrentOperation is set")
}
specParameters = instance.Status.InProgressProperties.InlineParameters
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question.

@nilebox
Copy link
Contributor Author

nilebox commented Apr 5, 2018

The alternative fix is to always read parameters from the spec, but also update InProgressProperties value every time to guarantee it didn't become stale.

Then, to fix the update conflict issue @kibbles-n-bytes wrote about, we could do GET before updating ExternalProperties to reduce the probability of conflict. And if the conflict still occurs, it might be fine that we never record ExternalProperties, and start reconciling a new spec. The status is never guaranteed to be 100% in-sync with OSB broker anyway, given that OSB doesn't have a mandatory GET instance endpoint.

This seems like an easier fix that doesn't require adding InlineProperties to status.

@nilebox
Copy link
Contributor Author

nilebox commented Apr 5, 2018

Closing, as #1916 seems to be simpler and good enough.

@nilebox nilebox closed this Apr 5, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants