Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPAM controllers get stuck in processing subnet which starts with IP address like 0.X.X.X #340

Closed
sjtufl opened this issue Nov 17, 2022 · 1 comment · Fixed by #342
Closed
Labels
bug Something isn't working

Comments

@sjtufl
Copy link
Collaborator

sjtufl commented Nov 17, 2022

Bug Report

Type: bug report

What happened

Create a new subnet which starts with IP address like 0.0.0.X, e.g.,

apiVersion: networking.alibaba.com/v1
kind: Subnet
metadata:
  name: testsn
spec:
  network: init
  range:
    cidr: 0.0.0.0/24
    gateway: ""
    version: "4"
    start: "0.0.0.2"
    end: "0.0.0.3"

Its status (usage) never gets updated.
By checking metrics of manager, we found that IPAM controllers seems get stuck in processing certain item.

# HELP workqueue_unfinished_work_seconds How many seconds of work has been done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
# TYPE workqueue_unfinished_work_seconds gauge
workqueue_unfinished_work_seconds{name="GlobalService"} 0
workqueue_unfinished_work_seconds{name="IPAM"} 80509.583495458
workqueue_unfinished_work_seconds{name="IPInstance"} 0
workqueue_unfinished_work_seconds{name="NetworkStatus"} 80509.584626613
workqueue_unfinished_work_seconds{name="Node"} 0
workqueue_unfinished_work_seconds{name="Pod"} 79452.589502282
workqueue_unfinished_work_seconds{name="Quota"} 0
workqueue_unfinished_work_seconds{name="RemoteCluster"} 0
workqueue_unfinished_work_seconds{name="RemoteClusterStatus"} 0
workqueue_unfinished_work_seconds{name="RemoteClusterUUID"} 0
workqueue_unfinished_work_seconds{name="SubnetStatus"} 80509.584568096

What you expected to happen

Usage of newly created subnet can be synced successfully.

How to reproduce it (as minimally and precisely as possible)

Anything else we need to know?

Environment

  • hybridnet version:
    v0.7.5
  • OS (e.g. cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Kubernetes version:
  • Install tools:
  • Others:
@sjtufl sjtufl added the bug Something isn't working label Nov 17, 2022
@mars1024 mars1024 changed the title IPAM controllers get stuck in processing subnet which starts with IP address like 0.0.0.X IPAM controllers get stuck in processing subnet which starts with IP address like 0.X.X.X Nov 17, 2022
@mars1024
Copy link
Collaborator

mars1024 commented Nov 17, 2022

try to fix referred package by containernetworking/plugins#782
/cc @sjtufl @hhyasdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants