Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scaling events mid-update may not match Deployment behavior #738

Open
jessesuen opened this issue Sep 26, 2020 · 1 comment
Open

Scaling events mid-update may not match Deployment behavior #738

jessesuen opened this issue Sep 26, 2020 · 1 comment
Labels
bug Something isn't working no-issue-activity

Comments

@jessesuen
Copy link
Member

jessesuen commented Sep 26, 2020

Summary

Rollouts scale differently than a Deployment, when the scale event happens in the middle of an update:

To reproduce:

  1. Apply canary rollout with no steps and:
spec:
  replicas: 4
  strategy:
    canary:
      maxSurge: 2
      maxUnavailable: 0
  1. Update rollout with a new spec such that it will crashloop backoff (this allows us to halt progress of an update)
  2. Rollout will surge up, such that there are 6 total replicas (2 new, 4 old)
  3. kubectl scale the rollout to 8 replicas
  4. Rollout will then have 10 replicas (6 new, 4 old). When compared to the deployment in this scenario, it will have 3 new, 7 old.

The reason seems to be related to the fact that the canary weight is calculated as 100%, when we do not have any steps. After step 3, when we are at 6 replicas (2 new, 4 old), and then suddenly want to scale to 8 replicas, the controller calculates it can get to 8 replicas there by solely adding new versions of pods, whereas Deployment attempts to preserve the current scale ratio.

Also after step 5, if you try to scale the rollout back down to 4, it doesn't do anything, which is another bug.

NOTE: the scaling logic may need to remain fundamentally different than Deployment since Rollout has a concept of a "stable" ReplicaSet which it decides to balance weights against. For example, when applying a revision 3 manifest in this scenario (after step 5), unlike Deployment, rollouts wants to balance the canary weights against revision 1 and 3.

Diagnostics

What version of Argo Rollouts are you running? v0.8.3


Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

@jessesuen jessesuen added the bug Something isn't working label Sep 26, 2020
@github-actions
Copy link
Contributor

github-actions bot commented Dec 7, 2022

This issue is stale because it has been open 60 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working no-issue-activity
Projects
None yet
Development

No branches or pull requests

1 participant