Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ZoneInsights subscription are invalid #7535

Closed
lahabana opened this issue Aug 17, 2023 · 3 comments
Closed

ZoneInsights subscription are invalid #7535

lahabana opened this issue Aug 17, 2023 · 3 comments
Assignees
@michaelbeaumont
Labels
area/multizone kind/bug A bug triage/accepted The issue was reviewed and is complete enough to start working on it

Comments

@lahabana
Copy link
Contributor

lahabana commented Aug 17, 2023 •
edited
Loading

What happened?

Multiple weird things here:

  • not ordered by connectTime (the newest connection should always be last)
  • a lot of non disconnected subscriptions seems to exist.

I've checked other insights and they don't seem to have this problem.

trimmed version using pbpaste | jq '.zoneInsight.subscriptions | del(.[] | .status.stat) | del(.[] | .config) | del(.[] | .version)'

[
  {
    "id": "ef2f9da4-9b50-4b7f-9530-3efb897d9ebb",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-16T15:46:54.968654235Z",
    "status": {
      "lastUpdateTime": "2023-08-16T15:46:56.242911919Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 383
  },
  {
    "id": "31f30740-bc80-4811-a262-4fcf467237b3",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-17T02:09:47.353040574Z",
    "status": {
      "lastUpdateTime": "2023-08-17T02:09:47.853503539Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 133
  },
  {
    "id": "c0cd81d9-00fa-4c36-9b98-9c1d43fc6a57",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-16T17:09:12.650835657Z",
    "status": {
      "lastUpdateTime": "2023-08-16T17:09:13.281269475Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 350
  },
  {
    "id": "6a67e101-b92c-4256-8ab9-1372534c6055",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-16T17:57:08.027177564Z",
    "status": {
      "lastUpdateTime": "2023-08-16T17:57:08.576920005Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 330
  },
  {
    "id": "88aa4963-5a7d-4bcf-9b7e-3612d5d8919c",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-17T03:58:56.543362277Z",
    "status": {
      "lastUpdateTime": "2023-08-17T03:58:57.020392357Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 90
  },
  {
    "id": "a2ce13f7-d28f-4cab-adab-59a1fa46022f",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-17T05:04:30.305811060Z",
    "disconnectTime": "2023-08-17T05:21:11.798920035Z",
    "status": {
      "lastUpdateTime": "2023-08-17T05:10:16.240163448Z",
      "total": {
        "responsesSent": "9"
      }
    },
    "generation": 4
  },
  {
    "id": "5891e6db-cd9e-46f2-b4fe-2f3b805d76b6",
    "globalInstanceId": "mink-control-plane-57cdd67669-bcknf-0676",
    "connectTime": "2023-08-17T06:19:04.295324637Z",
    "status": {
      "lastUpdateTime": "2023-08-17T06:19:05.577959060Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 34
  },
  {
    "id": "4f4c66d6-c236-49f1-8181-f32adc887a17",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-16T20:47:24.425755529Z",
    "status": {
      "lastUpdateTime": "2023-08-16T20:47:25.021264998Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 262
  },
  {
    "id": "2b9e341c-264b-4af4-bd64-0b00873985be",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-17T07:15:51.868865656Z",
    "status": {
      "lastUpdateTime": "2023-08-17T07:15:52.568221582Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 11
  },
  {
    "id": "f491c00d-dd66-488c-87dd-d456d87ed15e",
    "globalInstanceId": "mink-control-plane-57cdd67669-bz6n9-9bd1",
    "connectTime": "2023-08-17T00:06:58.387319166Z",
    "status": {
      "lastUpdateTime": "2023-08-17T00:06:58.934255144Z",
      "total": {
        "responsesSent": "37",
        "responsesAcknowledged": "37"
      }
    },
    "generation": 183
  }
]

full version:
insights.txt
@lahabana lahabana added area/multizone triage/pending This issue will be looked at on the next triage meeting kind/bug A bug labels Aug 17, 2023
@jakubdyszkiewicz jakubdyszkiewicz added triage/needs-reproducing Someone else should try to reproduce this and removed triage/pending This issue will be looked at on the next triage meeting labels Aug 21, 2023
@lukidzi
Copy link
Contributor

lukidzi commented Oct 4, 2023

Response
resp.txt

@jakubdyszkiewicz jakubdyszkiewicz added triage/accepted The issue was reviewed and is complete enough to start working on it and removed triage/needs-reproducing Someone else should try to reproduce this labels Oct 18, 2023
@jakubdyszkiewicz
Copy link
Contributor

Finalizer is executing IsOnline checking all subscriptions, but it only finishes the last one. It should finish all of them.

This was referenced Oct 25, 2023
Closed
Merged
@lahabana
Copy link
Contributor Author

lahabana commented Nov 8, 2023

Still saw an occurrence of this:

{
 "type": "ZoneOverview",
 "name": "local-laptop",
 "creationTime": "2023-11-06T15:49:24.031706Z",
 "modificationTime": "2023-11-06T15:49:24.031706Z",
 "zone": {},
 "zoneInsight": {
  "subscriptions": [
   {
    "id": "3eaef944-cc75-4bd7-8646-a13148a849b2",
    "globalInstanceId": "mink-control-plane-584857f855-n46jw-a44b",
    "connectTime": "2023-11-07T13:19:03.469975162Z",
    "disconnectTime": "2023-11-07T15:32:11.781341616Z",
    "status": {
     "lastUpdateTime": "2023-11-07T13:19:08.635644160Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 53
   },
   {
    "id": "73a68c18-1341-42a6-b556-bf5af9266ae6",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-98kv6-942a",
    "connectTime": "2023-11-07T15:32:17.522136293Z",
    "status": {
     "lastUpdateTime": "2023-11-07T15:32:22.743109033Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 438
   },
   {
    "id": "5bc0e324-5799-4a3c-9c30-f4b6c6e4e676",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-9qxtz-3e7a",
    "connectTime": "2023-11-07T18:30:22.423323824Z",
    "disconnectTime": "2023-11-08T08:33:09.576971013Z",
    "status": {
     "lastUpdateTime": "2023-11-07T18:30:27.689075949Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 337
   },
   {
    "id": "8d447ab0-8979-4172-8770-0bdd98996100",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-98kv6-942a",
    "connectTime": "2023-11-07T18:48:43.797928129Z",
    "status": {
     "lastUpdateTime": "2023-11-07T18:48:44.568591723Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 360
   },
   {
    "id": "7d949f39-c99d-46b2-85d7-cf559724a7cb",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-9qxtz-3e7a",
    "connectTime": "2023-11-07T22:31:20.574407613Z",
    "status": {
     "lastUpdateTime": "2023-11-07T22:31:21.222842122Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 271
   },
   {
    "id": "3beaae4f-47a3-4d02-b87b-9734eed1fd20",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-9qxtz-3e7a",
    "connectTime": "2023-11-07T22:48:44.576777535Z",
    "disconnectTime": "2023-11-08T08:33:09.608364796Z",
    "status": {
     "lastUpdateTime": "2023-11-07T22:48:45.350422247Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 233
   },
   {
    "id": "4e726f09-a87f-4430-a83d-b5ff01577f6d",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-9qxtz-3e7a",
    "connectTime": "2023-11-08T05:15:56.771840217Z",
    "status": {
     "lastUpdateTime": "2023-11-08T05:15:57.378306993Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 109
   },
   {
    "id": "db0c6e78-eb79-42b8-864f-e863b830545c",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-98kv6-942a",
    "connectTime": "2023-11-08T05:36:29.697945374Z",
    "disconnectTime": "2023-11-08T08:37:08.942746385Z",
    "status": {
     "lastUpdateTime": "2023-11-08T05:36:30.480633650Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 70
   },
   {
    "id": "c37491d7-2fb9-4131-8cd7-60187a738c7a",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-9qxtz-3e7a",
    "connectTime": "2023-11-08T05:52:56.784025008Z",
    "disconnectTime": "2023-11-08T06:00:50.274091780Z",
    "status": {
     "lastUpdateTime": "2023-11-08T05:52:57.394882860Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     }
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 3
   },
   {
    "id": "29dbe474-fb1e-40a8-9192-ee9b3128a6d8",
    "globalInstanceId": "mink-control-plane-86d6c5ccb-98kv6-942a",
    "connectTime": "2023-11-08T06:00:55.390412683Z",
    "disconnectTime": "2023-11-08T08:08:43.617292580Z",
    "status": {
     "lastUpdateTime": "2023-11-08T06:00:56.091056802Z",
     "total": {
      "responsesSent": "37",
      "responsesAcknowledged": "37"
     },
     "stat": "<<TRUNCATED>>",
    },
    "version": {
     "kumaCp": {
      "version": "2.4.3",
      "gitTag": "2.4.3",
      "gitCommit": "d0b2f9f264343a16ed4138d65605a0e978d9a6a7",
      "buildDate": "2023-10-11T22:00:24Z",
      "kumaCpGlobalCompatible": true
     }
    },
    "generation": 51,
    "config": "<<TRUNCATED>>"
  ],
  "envoyAdminStreams": {
   "configDumpGlobalInstanceId": "mink-control-plane-86d6c5ccb-98kv6-942a",
   "statsGlobalInstanceId": "mink-control-plane-86d6c5ccb-9qxtz-3e7a",
   "clustersGlobalInstanceId": "mink-control-plane-86d6c5ccb-9qxtz-3e7a"
  }
 }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michaelbeaumont michaelbeaumont

Labels
area/multizone kind/bug A bug triage/accepted The issue was reviewed and is complete enough to start working on it
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lahabana @michaelbeaumont @jakubdyszkiewicz @lukidzi