Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cant Retry Aborted Canary #3120

Closed
2 tasks done
wmiller112 opened this issue Oct 21, 2023 · 2 comments
Closed
2 tasks done

Cant Retry Aborted Canary #3120

wmiller112 opened this issue Oct 21, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@wmiller112
Copy link

wmiller112 commented Oct 21, 2023 •
edited
Loading

Checklist:

  • I've included steps to reproduce the bug.
  • I've included the version of argo rollouts.

Describe the bug

Starting with version 1.6.0, aborted rollouts get stuck when retried, not scaling up the canary replicaset. I've been experiencing this issue about 75% of the time for aborted rollouts, but haven't been able to figure out why it works as expected the other 25%. I cannot replicate on v1.5.1.

To Reproduce

  1. Configure a rollout with a canary strategy and an analysis step
    • Analysis is a simple datadog metric
    strategy:
    canary:
      steps:
        - setWeight: 20
        - analysis:
            templates:
              - templateName: error-rate
                clusterScope: true
            args:
              - name: stable-hash
                valueFrom:
                  podTemplateHashValue: Stable
              - name: canary-hash
                valueFrom:
                  podTemplateHashValue: Latest
  2. Update the image
  3. Abort the rollout while analysis is running
  4. Wait for the canary replicaset to scale down
  5. Retry the rollout

Expected behavior

The rollout would scale up the canary replicaset and start the rollout steps from the beginning.

Version

1.6.0

Logs

time="2023-10-21T00:33:49Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276614330\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutAddedToInformer' Rollout resource added to informer: example-project-staging/example-project"
time="2023-10-21T00:33:49Z" level=info msg="Rollout resource added to informer: example-project-staging/example-project" event_reason=RolloutAddedToInformer namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620418\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutNotCompleted' Rollout not completed, started update to revision 13 (7ddf848555)"
time="2023-10-21T00:37:47Z" level=info msg="Rollout not completed, started update to revision 13 (7ddf848555)" event_reason=RolloutNotCompleted namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Patched: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-10-18T19:58:12Z\",\"lastUpdateTime\":\"2023-10-18T19:58:12Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-10-20T13:12:15Z\",\"lastUpdateTime\":\"2023-10-20T13:12:15Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-10-21T00:24:58Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"Rollout \\\"example-project\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"}],\"currentPodHash\":\"7ddf848555\",\"currentStepIndex\":0,\"message\":\"more replicas need to be updated\",\"observedGeneration\":\"24\",\"phase\":\"Progressing\",\"updatedReplicas\":null}}" generation=24 namespace=example-project-staging resourceVersion=276620418 rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Updating replica set 'example-project-7ddf848555' revision from 0 to 13" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:37:47Z" level=info msg="Created ReplicaSet example-project-7ddf848555" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620431\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutUpdated' Rollout updated to revision 13"
time="2023-10-21T00:37:47Z" level=info msg="Rollout updated to revision 13" event_reason=RolloutUpdated namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620431\", FieldPath:\"\"}): type: 'Normal' reason: 'NewReplicaSetCreated' Created ReplicaSet example-project-7ddf848555 (revision 13)"
time="2023-10-21T00:37:47Z" level=info msg="Created ReplicaSet example-project-7ddf848555 (revision 13)" event_reason=NewReplicaSetCreated namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Set rollout condition: &RolloutCondition{Type:Progressing,Status:True,LastUpdateTime:2023-10-21 00:37:47.680264227 +0000 UTC m=+255.655168336,LastTransitionTime:2023-10-21 00:37:47.680264315 +0000 UTC m=+255.655168412,Reason:NewReplicaSetCreated,Message:Created new replica set \"example-project-7ddf848555\",}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620435\", FieldPath:\"\"}): type: 'Normal' reason: 'UpdatedDestinationRule' DestinationRule example-project subset updated (canary: 7ddf848555, stable: 5c57776cf9)"
time="2023-10-21T00:37:47Z" level=info msg="DestinationRule example-project subset updated (canary: 7ddf848555, stable: 5c57776cf9)" event_reason=UpdatedDestinationRule namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Previous weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:5c57776cf9,},Stable:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:7ddf848555,},Stable:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620435\", FieldPath:\"\"}): type: 'Normal' reason: 'TrafficWeightUpdated' Traffic weight updated "
time="2023-10-21T00:37:47Z" level=info msg="Traffic weight updated " event_reason=TrafficWeightUpdated namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"example-project-7ddf848555\": the object has been modified; please apply your changes to the latest version and try again" generation=24 namespace=example-project-staging resourceVersion=276620422 rollout=example-project
time="2023-10-21T00:37:47Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"example-project-7ddf848555\": the object has been modified; please apply your changes to the latest version and try again" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="rollout syncHandler queue retries: 6 : key \"example-project-staging/example-project\"" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Previous weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:5c57776cf9,},Stable:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:7ddf848555,},Stable:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620435\", FieldPath:\"\"}): type: 'Normal' reason: 'TrafficWeightUpdated' Traffic weight updated "
time="2023-10-21T00:37:47Z" level=info msg="Traffic weight updated " event_reason=TrafficWeightUpdated namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:37:47Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620435\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet example-project-7ddf848555 (revision 13) from 0 to 1"
time="2023-10-21T00:37:47Z" level=info msg="Scaled up ReplicaSet example-project-7ddf848555 (revision 13) from 0 to 1" event_reason=ScalingReplicaSet namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Not finished reconciling new ReplicaSet 'example-project-7ddf848555'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Not finished reconciling ReplicaSets" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Timed out (false) [last progress check: 2023-10-21 00:37:47 +0000 UTC - now: 2023-10-21 00:37:47.732488491 +0000 UTC m=+255.707392598]" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Patched: {\"status\":{\"canary\":{\"weights\":{\"canary\":{\"podTemplateHash\":\"7ddf848555\"}}},\"observedGeneration\":\"25\"}}" generation=25 namespace=example-project-staging resourceVersion=276620435 rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Timed out (false) [last progress check: 2023-10-21 00:37:47 +0000 UTC - now: 2023-10-21 00:37:47.755345154 +0000 UTC m=+255.730249265]" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Timed out (false) [last progress check: 2023-10-21 00:37:47 +0000 UTC - now: 2023-10-21 00:37:47.760369335 +0000 UTC m=+255.735273450]" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:37:47Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Timed out (false) [last progress check: 2023-10-21 00:37:47 +0000 UTC - now: 2023-10-21 00:37:47.784306462 +0000 UTC m=+255.759210576]" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:37:47Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":3,\"conditions\":[{\"lastTransitionTime\":\"2023-10-18T19:58:12Z\",\"lastUpdateTime\":\"2023-10-18T19:58:12Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-10-20T13:12:15Z\",\"lastUpdateTime\":\"2023-10-20T13:12:15Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-10-21T00:24:58Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"ReplicaSet \\\"example-project-7ddf848555\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"replicas\":3,\"updatedReplicas\":1}}" generation=25 namespace=example-project-staging resourceVersion=276620452 rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Timed out (false) [last progress check: 2023-10-21 00:37:47 +0000 UTC - now: 2023-10-21 00:37:47.83098697 +0000 UTC m=+255.805891076]" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:37:47Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:38:08Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620462\", FieldPath:\"\"}): type: 'Normal' reason: 'Updated VirtualService' VirtualService `example-project` set to desiredWeight '20'"
time="2023-10-21T00:38:08Z" level=info msg="VirtualService `example-project` set to desiredWeight '20'" event_reason="Updated VirtualService" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Previous weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:7ddf848555,},Stable:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:20,ServiceName:,PodTemplateHash:7ddf848555,},Stable:WeightDestination{Weight:80,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Traffic weight updated from 0 to 20" event_reason=TrafficWeightUpdated namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620462\", FieldPath:\"\"}): type: 'Normal' reason: 'TrafficWeightUpdated' Traffic weight updated from 0 to 20"
time="2023-10-21T00:38:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620462\", FieldPath:\"\"}): type: 'Normal' reason: 'RolloutStepCompleted' Rollout step 1/2 completed (setWeight: 20)"
time="2023-10-21T00:38:08Z" level=info msg="Rollout step 1/2 completed (setWeight: 20)" event_reason=RolloutStepCompleted namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":3,\"canary\":{\"weights\":{\"canary\":{\"weight\":20},\"stable\":{\"weight\":80}}},\"conditions\":[{\"lastTransitionTime\":\"2023-10-18T19:58:12Z\",\"lastUpdateTime\":\"2023-10-18T19:58:12Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-10-20T13:12:15Z\",\"lastUpdateTime\":\"2023-10-20T13:12:15Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-10-21T00:24:58Z\",\"lastUpdateTime\":\"2023-10-21T00:38:08Z\",\"message\":\"ReplicaSet \\\"example-project-7ddf848555\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"}],\"currentStepIndex\":1,\"readyReplicas\":3}}" generation=25 namespace=example-project-staging resourceVersion=276620462 rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Reconciling analysis step (stepIndex: 1)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Created AnalysisRun for step '1'" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555-13-1: Rollout example-project-staging/example-project"
time="2023-10-21T00:38:08Z" level=info msg="Started syncing Analysis at (2023-10-21 00:38:08.222287662 +0000 UTC m=+276.197191837)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Running initial measurement" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Taking 1 Measurement(s)..." analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Patched: {\"status\":{\"canary\":{\"currentStepAnalysisRunStatus\":{\"name\":\"example-project-7ddf848555-13-1\",\"status\":\"\"}}}}" generation=25 namespace=example-project-staging resourceVersion=276620773 rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Reconciling analysis step (stepIndex: 1)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Reconciling analysis step (stepIndex: 1)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Measurement Completed. Result: Successful" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Enqueueing analysis after 29.999920949s" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Patch status successfully" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Started syncing Analysis at (2023-10-21 00:38:08.627413555 +0000 UTC m=+276.602317649)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555-13-1: Rollout example-project-staging/example-project"
time="2023-10-21T00:38:08Z" level=info msg="Taking 0 Measurement(s)..." analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Enqueueing analysis after 29.372158042s" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:08Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Reconciling analysis step (stepIndex: 1)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:08Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276620781\", FieldPath:\"\"}): type: 'Normal' reason: 'AnalysisRunRunning' Step Analysis Run 'example-project-7ddf848555-13-1' Status New: 'Running' Previous: ''"
time="2023-10-21T00:38:09Z" level=info msg="Step Analysis Run 'example-project-7ddf848555-13-1' Status New: 'Running' Previous: ''" event_reason=AnalysisRunRunning namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:09Z" level=info msg="Patched: {\"status\":{\"canary\":{\"currentStepAnalysisRunStatus\":{\"status\":\"Running\"}}}}" generation=25 namespace=example-project-staging resourceVersion=276620781 rollout=example-project
time="2023-10-21T00:38:09Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:09Z" level=info msg="Reconciling analysis step (stepIndex: 1)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:38:38Z" level=info msg="Started syncing Analysis at (2023-10-21 00:38:38.000403268 +0000 UTC m=+305.975307396)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Running overdue measurement" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Taking 1 Measurement(s)..." analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Measurement Completed. Result: Successful" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Enqueueing analysis after 29.999891644s" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Patch status successfully" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Started syncing Analysis at (2023-10-21 00:38:38.238079644 +0000 UTC m=+306.212983827)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Taking 0 Measurement(s)..." analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:38:38Z" level=info msg="Enqueueing analysis after 29.761656911s" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Started syncing Analysis at (2023-10-21 00:39:08.000181438 +0000 UTC m=+335.975085614)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Running overdue measurement" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Taking 1 Measurement(s)..." analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Measurement Completed. Result: Failed" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Enqueueing analysis after 29.999857455s" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Patch status successfully" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Started syncing Analysis at (2023-10-21 00:39:08.192273991 +0000 UTC m=+336.167178177)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Taking 0 Measurement(s)..." analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:08Z" level=info msg="Enqueueing analysis after 29.807450721s" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="VirtualService `example-project` set to desiredWeight '0'" event_reason="Updated VirtualService" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="Previous weights: &TrafficWeights{Canary:WeightDestination{Weight:20,ServiceName:,PodTemplateHash:7ddf848555,},Stable:WeightDestination{Weight:80,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="New weights: &TrafficWeights{Canary:WeightDestination{Weight:0,ServiceName:,PodTemplateHash:7ddf848555,},Stable:WeightDestination{Weight:100,ServiceName:,PodTemplateHash:5c57776cf9,},Additional:[]WeightDestination{},Verified:nil,}" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="Traffic weight updated from 20 to 0" event_reason=TrafficWeightUpdated namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="Canceling the analysis run 'example-project-7ddf848555-13-1'" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276621829\", FieldPath:\"\"}): type: 'Normal' reason: 'Updated VirtualService' VirtualService `example-project` set to desiredWeight '0'"
time="2023-10-21T00:39:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276621829\", FieldPath:\"\"}): type: 'Normal' reason: 'TrafficWeightUpdated' Traffic weight updated from 20 to 0"
time="2023-10-21T00:39:36Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="Started syncing Analysis at (2023-10-21 00:39:36.882789826 +0000 UTC m=+364.857693987)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Skipping measurement: run is terminating" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Taking 0 Measurement(s)..." analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Metric 'error-rate' transitioned from Running -> Successful" analysisrun=example-project-7ddf848555-13-1 metric=error-rate namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Metric 'error-rate' Completed. Result: Successful" analysisrun=example-project-7ddf848555-13-1 event_reason=MetricSuccessful namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"AnalysisRun\", Namespace:\"example-project-staging\", Name:\"example-project-7ddf848555-13-1\", UID:\"b43fdf32-43a9-45db-8099-48b003679f65\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276621831\", FieldPath:\"\"}): type: 'Normal' reason: 'MetricSuccessful' Metric 'error-rate' Completed. Result: Successful"
time="2023-10-21T00:39:36Z" level=info msg="Analysis Completed. Result: Successful" analysisrun=example-project-7ddf848555-13-1 event_reason=AnalysisRunSuccessful namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"AnalysisRun\", Namespace:\"example-project-staging\", Name:\"example-project-7ddf848555-13-1\", UID:\"b43fdf32-43a9-45db-8099-48b003679f65\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276621831\", FieldPath:\"\"}): type: 'Normal' reason: 'AnalysisRunSuccessful' Analysis Completed. Result: Successful"
time="2023-10-21T00:39:36Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:39:36Z" level=info msg="Patch status successfully" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Started syncing Analysis at (2023-10-21 00:39:36.899720849 +0000 UTC m=+364.874625016)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:36Z" level=info msg="Set 'scale-down-deadline' annotation on 'example-project-7ddf848555' to 2023-10-21T00:40:06Z (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:36Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555-13-1: Rollout example-project-staging/example-project"
time="2023-10-21T00:39:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276621829\", FieldPath:\"\"}): type: 'Warning' reason: 'RolloutAborted' Rollout aborted update to revision 13"
time="2023-10-21T00:39:38Z" level=info msg="Started syncing Analysis at (2023-10-21 00:39:38.000577771 +0000 UTC m=+365.975481955)" analysisrun=example-project-7ddf848555-13-1 namespace=example-project-staging
time="2023-10-21T00:39:38Z" level=warning msg="Rollout aborted update to revision 13" event_reason=RolloutAborted namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Patched: {\"status\":{\"abortedAt\":\"2023-10-21T00:39:38Z\",\"canary\":{\"weights\":{\"canary\":{\"weight\":0},\"stable\":{\"weight\":100}}},\"conditions\":[{\"lastTransitionTime\":\"2023-10-18T19:58:12Z\",\"lastUpdateTime\":\"2023-10-18T19:58:12Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-10-20T13:12:15Z\",\"lastUpdateTime\":\"2023-10-20T13:12:15Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-10-21T00:39:36Z\",\"lastUpdateTime\":\"2023-10-21T00:39:36Z\",\"message\":\"Rollout aborted update to revision 13\",\"reason\":\"RolloutAborted\",\"status\":\"False\",\"type\":\"Progressing\"}],\"currentStepIndex\":0,\"message\":\"RolloutAborted: Rollout aborted update to revision 13\",\"phase\":\"Degraded\"}}" generation=25 namespace=example-project-staging resourceVersion=276621829 rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="RS 'example-project-7ddf848555' has not reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Patched: {\"status\":{\"canary\":{\"currentStepAnalysisRunStatus\":{\"message\":\"Run Terminated\",\"status\":\"Successful\"}}}}" generation=25 namespace=example-project-staging resourceVersion=276621861 rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="RS 'example-project-7ddf848555' has not reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:39:38Z" level=info msg="RS 'example-project-7ddf848555' has not reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="RS 'example-project-7ddf848555' has reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:40:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"example-project-staging\", Name:\"example-project\", UID:\"a701dd41-3496-44c3-975a-c9b5c1a5be9d\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"276621862\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet example-project-7ddf848555 (revision 13) from 1 to 0"
time="2023-10-21T00:40:06Z" level=info msg="Scaled down ReplicaSet example-project-7ddf848555 (revision 13) from 1 to 0" event_reason=ScalingReplicaSet namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Not finished reconciling new ReplicaSet 'example-project-7ddf848555'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Not finished reconciling ReplicaSets" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="RS 'example-project-7ddf848555' has reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:40:06Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="RS 'example-project-7ddf848555' has reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:40:06Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="RS 'example-project-7ddf848555' has reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Patched: {\"status\":{\"HPAReplicas\":2,\"availableReplicas\":2,\"readyReplicas\":2,\"replicas\":2,\"updatedReplicas\":null}}" generation=25 namespace=example-project-staging resourceVersion=276621862 rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="Scale down new rs 'example-project-7ddf848555' on abort (30s)" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="New rs 'example-project-7ddf848555' has scaledown deadline annotation: 2023-10-21T00:40:06Z" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:06Z" level=info msg="RS 'example-project-7ddf848555' has reached the scaleDownTime" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=info msg="Patched conditions: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-10-18T19:58:12Z\",\"lastUpdateTime\":\"2023-10-18T19:58:12Z\",\"message\":\"Rollout is paused\",\"reason\":\"RolloutPaused\",\"status\":\"False\",\"type\":\"Paused\"},{\"lastTransitionTime\":\"2023-10-20T13:12:15Z\",\"lastUpdateTime\":\"2023-10-20T13:12:15Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2023-10-21T00:37:47Z\",\"lastUpdateTime\":\"2023-10-21T00:37:47Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2023-10-21T00:40:38Z\",\"lastUpdateTime\":\"2023-10-21T00:40:38Z\",\"message\":\"Retrying Rollout after abort\",\"reason\":\"RolloutRetry\",\"status\":\"Unknown\",\"type\":\"Progressing\"}],\"message\":\"more replicas need to be updated\",\"phase\":\"Progressing\"}}" generation=25 namespace=example-project-staging resourceVersion=276622799 rollout=example-project
time="2023-10-21T00:40:38Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=info msg="Skipping analysis: isAborted: false, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=info msg="Enqueueing parent of example-project-staging/example-project-7ddf848555: Rollout example-project-staging/example-project"
time="2023-10-21T00:40:38Z" level=info msg="Removed 'scale-down-deadline' annotation from RS 'example-project-7ddf848555'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on replicasets.apps \"example-project-7ddf848555\": the object has been modified; please apply your changes to the latest version and try again" generation=25 namespace=example-project-staging resourceVersion=276622799 rollout=example-project
time="2023-10-21T00:40:38Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"example-project-7ddf848555\": the object has been modified; please apply your changes to the latest version and try again" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=info msg="rollout syncHandler queue retries: 4 : key \"example-project-staging/example-project\"" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=info msg="Reconciling TrafficRouting with type 'Istio'" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=info msg="Skipping analysis: isAborted: false, promoteFull: false, rollbackToScaleDownDelay: true, initialDeploy: false" namespace=example-project-staging rollout=example-project
time="2023-10-21T00:40:38Z" level=error msg="roCtx.reconcile err the server rejected our request due to an error in our request" generation=25 namespace=example-project-staging resourceVersion=276622800 rollout=example-project
time="2023-10-21T00:40:38Z" level=error msg="rollout syncHandler error: the server rejected our request due to an error in our request" namespace=example-project-staging rollout=example-project

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
@wmiller112 wmiller112 added the bug Something isn't working label Oct 21, 2023
@bpoland
Copy link
Contributor

bpoland commented Nov 8, 2023

Hey is this a dupe of #3080 ?

@wmiller112
Copy link
Author

Hey is this a dupe of #3080 ?

I believe it may be. Ill follow along there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bpoland @wmiller112