Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Traffic Manager crashes: Observed a panic: &runtime.TypeAssertionError #3149

Closed
xzhang-brex opened this issue Apr 24, 2023 · 1 comment · Fixed by #3153
Closed

Traffic Manager crashes: Observed a panic: &runtime.TypeAssertionError #3149

xzhang-brex opened this issue Apr 24, 2023 · 1 comment · Fixed by #3153
Assignees
@thallgren

Comments

@xzhang-brex
Copy link
Contributor

Describe the bug
The Traffic Manager crashes.

To Reproduce

  1. Enabled the debug log level for traffic manager
  2. Observe organic requests being processed by the traffic manager from k9s's log view (we have a huge cluster and we can see the debug log is scrolling down like crazy)
  3. ~10 minutes ish, we will see below panic, and the traffic manager crashes from there, and then restart by k8s
│ E0420 23:30:38.003782       1 runtime.go:79] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x31de940), concrete:(*runtime._type)(0x3376bc0), asserted:(*runtime._type)(0x35ea1e0), missingMethod:""} (interface conversion: interface {} is ca │
│ goroutine 2825 [running]:                                                                                                                                                                                                                                                  │
│ k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3290dc0?, 0xc016013110})                                                                                                                                                                                                  │
│     k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x99                                                                                                                                                                                                       │
│ k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x0?})                                                                                                                                                                                                         │
│     k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x75                                                                                                                                                                                                       │
│ panic({0x3290dc0, 0xc016013110})                                                                                                                                                                                                                                           │
│     runtime/panic.go:884 +0x213                                                                                                                                                                                                                                            │
│ github.com/telepresenceio/telepresence/v2/cmd/traffic/cmd/manager/internal/cluster.newPodWatcher.func2({0x3376bc0?, 0xc01436d640?})                                                                                                                                        │
│     github.com/telepresenceio/telepresence/v2/cmd/traffic/cmd/manager/internal/cluster/podwatcher.go:47 +0x5e                                                                                                                                                              │
│ k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(...)                                                                                                                                                                                                       │
│     k8s.io/[email protected]/tools/cache/controller.go:246                                                                                                                                                                                                                 │
│ k8s.io/client-go/tools/cache.(*processorListener).run.func1()                                                                                                                                                                                                              │
│     k8s.io/[email protected]/tools/cache/shared_informer.go:913 +0xaf                                                                                                                                                                                                      │
│ k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)                                                                                                                                                                                                                │
│     k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x3e                                                                                                                                                                                                            │
│ k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0000ccf38?, {0x3a0b4e0, 0xc045e79b00}, 0x1, 0xc0580bfb00)                                                                                                                                                                │
│     k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xb6                                                                                                                                                                                                            │
│ k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x120ed65?, 0x3b9aca00, 0x0, 0x78?, 0x29e84e6?)                                                                                                                                                                              │
│     k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x89                                                                                                                                                                                                            │
│ k8s.io/apimachinery/pkg/util/wait.Until(...)                                                                                                                                                                                                                               │
│     k8s.io/[email protected]/pkg/util/wait/wait.go:92                                                                                                                                                                                                                   │
│ k8s.io/client-go/tools/cache.(*processorListener).run(0xc05336ea80)                                                                                                                                                                                                        │
│     k8s.io/[email protected]/tools/cache/shared_informer.go:905 +0x6b                                                                                                                                                                                                      │
│ k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()                                                                                                                                                                                                                   │
│     k8s.io/[email protected]/pkg/util/wait/wait.go:75 +0x5a                                                                                                                                                                                                             │
│ created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start                                                                                                                                                                                                                │
│     k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x85                                                                                                                                                                                                             │
│ panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod [recovered]                                                                                                                                                                       │
│     panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Pod

Expected behavior
The traffic manager should not crash.

Versions (please complete the following information):
Client : v2.13.0
Root Daemon : v2.13.0
User Daemon : v2.13.0
Traffic Manager: v2.13.0

VPN-related bugs:
If you're reporting an issue around telepresence connectivity when using a VPN,
and are running Telepresence 2.4.8 or above, please also attach the output
of telepresence test-vpn, and the following information:

  • Which VPN client are you using?
  • Which VPN server are you using?
  • How is your VPN pushing DNS configuration? It may be useful to add the contents of /etc/resolv.conf

Additional context
Add any other context about the problem here.
@xzhang-brex
Copy link
Contributor Author

@jsalem-brex

@thallgren thallgren self-assigned this Apr 25, 2023
thallgren added a commit that referenced this issue Apr 28, 2023
@thallgren
Fix unchecked cast panic in traffic-manager pod watcher.
c36f33d 
A watcher isn't guaranteed to always receive an object of the watched
type. The `DeleteFunc` in particular, can receive an instance of
`cache.DeleteFinalStateUnknown`. This commit ensures that the traffic-
manager's node and pod watcher uses checked casts only and that the
`cache.DeleteFinalStateUnknown` is handled properly.

Closed #3149

Signed-off-by: Thomas Hallgren <[email protected]>
thallgren added a commit that referenced this issue Apr 28, 2023
@thallgren
Fix unchecked cast panic in traffic-manager pod watcher.
8f187f0 
A watcher isn't guaranteed to always receive an object of the watched
type. The `DeleteFunc` in particular, can receive an instance of
`cache.DeleteFinalStateUnknown`. This commit ensures that the traffic-
manager's node and pod watcher uses checked casts only and that the
`cache.DeleteFinalStateUnknown` is handled properly.

Closes #3149

Signed-off-by: Thomas Hallgren <[email protected]>
@thallgren thallgren mentioned this issue Apr 28, 2023
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thallgren thallgren

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix unchecked cast panic in traffic-manager pod watcher. telepresenceio/telepresence
2 participants
@thallgren @xzhang-brex