Pagination changes given new vai cache backed steve api (#11110)

* Changes for new design - New visuals - Pagination controls --> load more - finished testing of label select with pagination off # Conflicts: # shell/edit/provisioning.cattle.io.cluster/__tests__/Basics.tests.ts * Changes following review * Update Node list to support server-side pagination - Setup pagination headers for the node type - Define a pattern for fetching custom list secondary resources - Major improvements to the way pagination settings are defined and created - Lots of docs improvements - Handle calling fetch again once fetch is in progress (nuxt caches running request) - Validate filter fields (not all are supported by the vai cache - General pagination fixes * Lint / test / fixes * Improvements to configmap e2e test & Improve pagination disabled * Beef up validation * Fix missing name column in non-server-side paginated node list * Fix PR automation actions - fix syntax - catch scenario where a pr has no fixed issue > There's duplication between files, see #10534 * CI bump * Fixes post merge * Wire in 2.9.0 settings for server-side pagination - Everything is gated on `on-disk-steve-cache` feature flag - There's a backend in progress item to resolve a `revision` issue, until then disable watching a resource given it - Global Settings - Performance - Added new setting to enable server side pagination - this is incompatible with two other performance settings * Integrate pagination with configmaps in cis clusterscanbenchmark edit form Also - improved labeled select pagination - gate label select pagination functinality on steve cache being enabled * - harvester machine-config - project monitoring (and bug fixes) * Disable workload screen if vai cache is on - temp step until we get new overview * TODOs and TEST * Conditionally remove fetch of all secrets from SelectOrCreateAuthSecret * TODOs and TEST * Update SimpleSecretSelector - only used in monitoring.coreos.com.alertmanagerconfig context * View and Edit ingress - secrets * node detail page - pods list * Backup/Restore: Secrets (WIP) * Backup/Restore: Secrets, and other usages of SimpleSecretSelector / SelectOrCreateAuthSecret * Edit: Service account * Add comments for remaining items * Paginate Secret selection for logging providers - Allow `None` option in Paginationed LabelSelect - Optionally classify pagination response * WIP * fixes arfter merge * Don't suggest container names, not practical - previously all pods were fetched... and we scrapped all container names from them - this is a scaling nightmare, user now must just enter the name/s to match * Avoid findAll secrets in SimpleSecretSelector * tidying up * Move LabeledSelect/index.vue back to LabeledSelect.vue to not break extensions * changes after self review... 1 * changes after self review... 2 * ooof * changes after self review... 3 * fix formatting * Link new paginated label select with pagination setting * Work around failing kubewarden unit tests in check-plugins gate * Fix backup.spec e2e test * fix formatting, paginationUtils.isSteveCacheEnabled --> paginationUtils.isEnabled * Don't fetch all secrets on cloud creds page * Fix backup.spec e2e test * TODO tidying / tracking * don't getch ALL workloads for hacky way to get a link to a service's workload * Fix bad merge * Updates after working with vai cache image * test fixes * Create a convienence wrapper called ResourceLabelSelector that hides most of the complexity * fix unit test * Updates following review * Remove workload health until #10417 is resolved * Updates following review * changes following self review * Fix bottom bar of edit backup, edit restore pages * revert temp change * changes following self review * Workaround for kubewarden unit tests in check plugin gate * bump * Fix e2e * Fix linting * type fixing * - improve filtering without pagination - update allowed fields given latest backend changes - enable on by default exact filter string matches (disable for lists * remove temp code * fix linting * Changes following review * Fixes for vai cache feature flag - name was changed from on-disk-steve-cache to ui-sql-cache - fix timing issue - don't watch resources until we know the vai cache feature flag * Changes following review * Fix two sketchy tests - new exception in docs page - don't nav to page via button click and then goto same page * More test improvement - force user to go to tab which is source of route guard issue - move setup stuff to a test for cypress to re-retry
rancher · Jun 11, 2024 · aaf2d69 · aaf2d69
1 parent 76742f4
commit aaf2d69
Show file tree

Hide file tree

Showing 20 changed files with 197 additions and 93 deletions.
diff --git a/cypress/e2e/po/pages/extensions.po.ts b/cypress/e2e/po/pages/extensions.po.ts
@@ -54,8 +54,9 @@ export default class ExtensionsPagePo extends PagePo {
     // create a new clusterrepo
     const appRepoList = new RepositoriesPagePo('local', 'apps');
 
-    // appRepoList.waitForPage();
-    appRepoList.waitForGoTo('/v1/catalog.cattle.io.clusterrepos?exclude=metadata.managedFields');
+    appRepoList.waitForPage();
+    appRepoList.list().checkVisible();
+
     appRepoList.create();
 
     const appRepoCreate = new ChartRepositoriesCreateEditPo('local', 'apps');

diff --git a/cypress/e2e/tests/navigation/side-nav/main-side-menu.spec.ts b/cypress/e2e/tests/navigation/side-nav/main-side-menu.spec.ts
@@ -4,6 +4,7 @@ import PagePo from '@/cypress/e2e/po/pages/page.po';
 import ProductNavPo from '@/cypress/e2e/po/side-bars/product-side-nav.po';
 import ClusterManagerListPagePo from '@/cypress/e2e/po/pages/cluster-manager/cluster-manager-list.po';
 import { generateFakeClusterDataAndIntercepts } from '@/cypress/e2e/blueprints/nav/fake-cluster';
+import { RANCHER_PAGE_EXCEPTIONS, catchTargetPageException } from '@/cypress/support/utils/exception-utils';
 
 const longClusterDescription = 'this-is-some-really-really-really-really-really-really-long-decription';
 const fakeProvClusterId = 'some-fake-cluster-id';
@@ -47,6 +48,8 @@ describe('Side Menu: main', () => {
 
     // testing https://github.com/rancher/dashboard/issues/10192
     it('"documentation" link in editing a cluster should open in a new tab', { tags: ['@navigation', '@adminUser'] }, () => {
+      catchTargetPageException(RANCHER_PAGE_EXCEPTIONS);
+
       const page = new PagePo('');
       const clusterList = new ClusterManagerListPagePo('_');
 

diff --git a/cypress/e2e/tests/pages/extensions/extensions.spec.ts b/cypress/e2e/tests/pages/extensions/extensions.spec.ts
@@ -10,22 +10,22 @@ const UI_PLUGINS_PARTNERS_REPO_URL = 'https://github.com/rancher/partner-extensi
 const UI_PLUGINS_PARTNERS_REPO_NAME = 'partner-extensions';
 
 describe('Extensions page', { tags: ['@extensions', '@adminUser'] }, () => {
-  before(() => {
+  beforeEach(() => {
     cy.login();
+  });
 
+  it('add repository', () => {
+    // This should be in a `before` however is flaky. Move it to an `it` to let cypress retry
     const extensionsPo = new ExtensionsPagePo();
 
     extensionsPo.goTo();
     extensionsPo.waitForPage();
+    extensionsPo.extensionTabInstalledClick(); // Avoid nav guard failures that probably auto move user to this tab
 
     // install the rancher plugin examples
     extensionsPo.addExtensionsRepository('https://github.com/rancher/ui-plugin-examples', 'main', 'rancher-plugin-examples');
   });
 
-  beforeEach(() => {
-    cy.login();
-  });
-
   it('has the correct title', () => {
     const extensionsPo = new ExtensionsPagePo();
 

diff --git a/pkg/harvester-manager/machine-config/harvester.vue b/pkg/harvester-manager/machine-config/harvester.vue
@@ -40,7 +40,7 @@ import { stringify, exceptionToErrorsArray } from '@shell/utils/error';
 import { isValidMac } from '@shell/utils/validators/cidr';
 import { HCI as HCI_ANNOTATIONS, STORAGE } from '@shell/config/labels-annotations';
 import { isEqual } from 'lodash';
-import { PaginationArgs, PaginationFilterField, PaginationParamFilter } from '~/shell/types/store/pagination.types';
+import { FilterArgs, PaginationFilterField, PaginationParamFilter } from '@shell/types/store/pagination.types';
 
 const STORAGE_NETWORK = 'storage-network.settings.harvesterhci.io';
 
@@ -135,10 +135,8 @@ export default {
         let configMapsUrl = `${ url }/${ CONFIG_MAP }s`;
 
         if (this.$store.getters[`cluster/paginationEnabled`](CONFIG_MAP)) {
-          const pagination = new PaginationArgs({
-            page:     1,
-            pageSize: -1,
-            filters:  [
+          const pagination = new FilterArgs({
+            filters: [
               PaginationParamFilter.createMultipleFields([
                 new PaginationFilterField({ field: `metadata.label["${ HCI_ANNOTATIONS.CLOUD_INIT }"]`, value: 'user' }),
                 new PaginationFilterField({ field: `metadata.label["${ HCI_ANNOTATIONS.CLOUD_INIT }"]`, value: 'network' })

diff --git a/shell/components/form/SelectOrCreateAuthSecret.vue b/shell/components/form/SelectOrCreateAuthSecret.vue
@@ -8,7 +8,7 @@ import { base64Encode } from '@shell/utils/crypto';
 import { addObjects, insertAt } from '@shell/utils/array';
 import { sortBy } from '@shell/utils/sort';
 import {
-  PaginationArgs,
+  FilterArgs,
   PaginationFilterField,
   PaginationParamFilter,
 } from '@shell/types/store/pagination.types';
@@ -411,9 +411,8 @@ export default {
       const findPageArgs = {
         // Of type ActionFindPageArgs
         namespaced: this.filterByNamespace ? this.namespace : '',
-        pagination: new PaginationArgs({
-          pageSize: -1,
-          filters:  [
+        pagination: new FilterArgs({
+          filters: [
             PaginationParamFilter.createMultipleFields(
               this.secretTypes.map(
                 (t) => new PaginationFilterField({

diff --git a/shell/config/pagination-table-headers.js b/shell/config/pagination-table-headers.js
@@ -25,8 +25,8 @@ export const STEVE_STATE_COL = {
   ...STATE,
   // value:  'metadata.state.name', Use the state as defined by the resource rather than converted via the model.
   // This means we'll show something different to what we sort and filter on.
-  sort:   ['metadata.state.name'],
-  search: 'metadata.state.name',
+  sort:   [], // ['metadata.state.name'], // Pending API support
+  search: false, // 'metadata.state.name', // Pending API support
 };
 
 export const STEVE_AGE_COL = {

diff --git a/shell/detail/networking.k8s.io.ingress.vue b/shell/detail/networking.k8s.io.ingress.vue
@@ -6,7 +6,7 @@ import Rules from '@shell/edit/networking.k8s.io.ingress/Rules';
 import ResourceTabs from '@shell/components/form/ResourceTabs';
 import Tab from '@shell/components/Tabbed/Tab';
 import { SECRET_TYPES as TYPES } from '@shell/config/secret';
-import { PaginationArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
+import { FilterArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
 
 export default {
   name:       'CRUIngress',
@@ -25,9 +25,8 @@ export default {
     if (this.$store.getters[`cluster/paginationEnabled`](SECRET)) {
       const findPageArgs = { // Of type ActionFindPageArgs
         namespaced: this.value.metadata.namespace,
-        pagination: new PaginationArgs({
-          pageSize: -1,
-          filters:  PaginationParamFilter.createSingleField({
+        pagination: new FilterArgs({
+          filters: PaginationParamFilter.createSingleField({
             field: 'metadata.fields.1',
             value: TYPES.TLS
           })

diff --git a/shell/detail/node.vue b/shell/detail/node.vue
@@ -19,7 +19,7 @@ import { mapGetters } from 'vuex';
 import { allDashboardsExist } from '@shell/utils/grafana';
 import Loading from '@shell/components/Loading';
 import metricPoller from '@shell/mixins/metric-poller';
-import { PaginationArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
+import { FilterArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
 
 const NODE_METRICS_DETAIL_URL = '/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/d/rancher-node-detail-1/rancher-node-detail?orgId=1';
 const NODE_METRICS_SUMMARY_URL = '/api/v1/namespaces/cattle-monitoring-system/services/http:rancher-monitoring-grafana:80/proxy/d/rancher-node-1/rancher-node?orgId=1';
@@ -52,11 +52,9 @@ export default {
     if (this.filterByApi) {
       // Only get pods associated with this node. The actual values used are from a get all in node model `pods` getter (this works as it just gets all...)
       const opt = { // Of type ActionFindPageArgs
-        pagination: new PaginationArgs({
-          page:     -1,
-          pageSize: -1,
-          sort:     [{ field: 'metadata.name', asc: true }],
-          filters:  PaginationParamFilter.createSingleField({
+        pagination: new FilterArgs({
+          sort:    [{ field: 'metadata.name', asc: true }],
+          filters: PaginationParamFilter.createSingleField({
             field: 'spec.nodeName',
             value: this.value.id,
           })

diff --git a/shell/edit/networking.k8s.io.ingress/index.vue b/shell/edit/networking.k8s.io.ingress/index.vue
@@ -16,7 +16,7 @@ import Certificates from './Certificates';
 import Rules from './Rules';
 import IngressClass from './IngressClass';
 import Loading from '@shell/components/Loading';
-import { PaginationArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
+import { FilterArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
 
 export default {
   name:       'CRUIngress',
@@ -211,9 +211,8 @@ export default {
     filterSecretsByApi() {
       const findPageArgs = { // Of type ActionFindPageArgs
         namespaced: this.value.metadata.namespace,
-        pagination: new PaginationArgs({
-          pageSize: -1,
-          filters:  PaginationParamFilter.createSingleField({
+        pagination: new FilterArgs({
+          filters: PaginationParamFilter.createSingleField({
             field: 'metadata.fields.1',
             value: TYPES.TLS
           })

diff --git a/shell/edit/resources.cattle.io.backup.vue b/shell/edit/resources.cattle.io.backup.vue
@@ -17,7 +17,7 @@ import { NAMESPACE, _VIEW } from '@shell/config/query-params';
 import { sortBy } from '@shell/utils/sort';
 import { get } from '@shell/utils/object';
 import { formatEncryptionSecretNames } from '@shell/utils/formatter';
-import { PaginationArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
+import { FilterArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
 import { SECRET_TYPES } from '@shell/config/secret';
 
 export default {
@@ -65,9 +65,8 @@ export default {
     if (this.$store.getters[`cluster/paginationEnabled`](SECRET)) {
       const findPageArgs = { // Of type ActionFindPageArgs
         namespaced: this.chartNamespace,
-        pagination: new PaginationArgs({
-          pageSize: -1,
-          filters:  PaginationParamFilter.createSingleField({
+        pagination: new FilterArgs({
+          filters: PaginationParamFilter.createSingleField({
             field: 'metadata.fields.1',
             value: SECRET_TYPES.OPAQUE
           })

diff --git a/shell/edit/resources.cattle.io.restore.vue b/shell/edit/resources.cattle.io.restore.vue
@@ -14,7 +14,7 @@ import { allHash } from '@shell/utils/promise';
 import { get } from '@shell/utils/object';
 import { _CREATE } from '@shell/config/query-params';
 import { formatEncryptionSecretNames } from '@shell/utils/formatter';
-import { PaginationArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
+import { FilterArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
 import { SECRET_TYPES } from '@shell/config/secret';
 
 export default {
@@ -62,9 +62,8 @@ export default {
     if (this.$store.getters[`cluster/paginationEnabled`](SECRET)) {
       const findPageArgs = { // Of type ActionFindPageArgs
         namespaced: this.chartNamespace,
-        pagination: new PaginationArgs({
-          pageSize: -1,
-          filters:  PaginationParamFilter.createSingleField({
+        pagination: new FilterArgs({
+          filters: PaginationParamFilter.createSingleField({
             field: 'metadata.fields.1',
             value: SECRET_TYPES.OPAQUE
           })

diff --git a/shell/edit/serviceaccount.vue b/shell/edit/serviceaccount.vue
@@ -9,7 +9,7 @@ import { Checkbox } from '@components/Form/Checkbox';
 import { SECRET } from '@shell/config/types';
 import { TYPES as SECRET_TYPES } from '@shell/models/secret';
 import LabeledSelect from '@shell/components/form/LabeledSelect';
-import { PaginationArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
+import { FilterArgs, PaginationParamFilter } from '@shell/types/store/pagination.types';
 
 export default {
   name: 'ServiceAccount',
@@ -70,9 +70,8 @@ export default {
     filterSecretsByApi() {
       const findPageArgs = { // Of type ActionFindPageArgs
         namespaced: this.value.metadata.namespace,
-        pagination: new PaginationArgs({
-          pageSize: -1,
-          filters:  PaginationParamFilter.createMultipleFields(this.secretTypes.map((t) => ({
+        pagination: new FilterArgs({
+          filters: PaginationParamFilter.createMultipleFields(this.secretTypes.map((t) => ({
             field:  'metadata.fields.1',
             value:  t,
             equals: true

diff --git a/shell/list/node.vue b/shell/list/node.vue
@@ -9,7 +9,7 @@ import { CAPI as CAPI_ANNOTATIONS } from '@shell/config/labels-annotations.js';
 
 import { defineComponent } from 'vue';
 import { ActionFindPageArgs } from '@shell/types/store/dashboard-store.types';
-import { PaginationArgs, PaginationFilterField, PaginationParamFilter } from '@shell/types/store/pagination.types';
+import { FilterArgs, PaginationFilterField, PaginationParamFilter } from '@shell/types/store/pagination.types';
 
 import {
   CAPI,
@@ -147,8 +147,7 @@ export default defineComponent({
 
         const opt: ActionFindPageArgs = {
           force:      true,
-          pagination: new PaginationArgs({
-            page:    -1,
+          pagination: new FilterArgs({
             filters: new PaginationParamFilter({
               fields: this.rows.map((r: any) => new PaginationFilterField({
                 field: 'metadata.name',
@@ -220,8 +219,7 @@ export default defineComponent({
         // See https://github.com/rancher/dashboard/issues/10743
         const opt: ActionFindPageArgs = {
           force,
-          pagination: new PaginationArgs({
-            page:    -1,
+          pagination: new FilterArgs({
             filters: PaginationParamFilter.createMultipleFields(this.rows.map((r: any) => new PaginationFilterField({
               field: 'status.nodeName',
               value: r.id
@@ -242,8 +240,7 @@ export default defineComponent({
           const opt: ActionFindPageArgs = {
             force,
             namespaced: namespace,
-            pagination: new PaginationArgs({
-              page:    -1,
+            pagination: new FilterArgs({
               filters: PaginationParamFilter.createMultipleFields(
                 this.rows.reduce((res: PaginationFilterField[], r: any ) => {
                   const name = r.metadata?.annotations?.[CAPI_ANNOTATIONS.MACHINE_NAME];
@@ -269,8 +266,7 @@ export default defineComponent({
         // Note - fetching pods for current page could be a LOT still (probably max of 3k - 300 pods per node x 100 nodes in a page)
         const opt: ActionFindPageArgs = {
           force,
-          pagination: new PaginationArgs({
-            page:    -1,
+          pagination: new FilterArgs({
             filters: PaginationParamFilter.createMultipleFields(
               this.rows.map((r: any) => new PaginationFilterField({
                 field: 'spec.nodeName',

diff --git a/shell/list/workload.vue b/shell/list/workload.vue
@@ -4,6 +4,7 @@ import {
   WORKLOAD_TYPES, SCHEMA, NODE, POD, LIST_WORKLOAD_TYPES
 } from '@shell/config/types';
 import ResourceFetch from '@shell/mixins/resource-fetch';
+import { WORKLOAD_HEALTH_SCALE } from '@shell/config/table-headers';
 
 const schema = {
   id:         'workload',
@@ -93,6 +94,10 @@ export default {
       return this.$route.params.resource === schema.id;
     },
 
+    paginationEnabled() {
+      return !this.allTypes && this.$store.getters[`cluster/paginationEnabled`]();
+    },
+
     schema() {
       const { params:{ resource:type } } = this.$route;
 
@@ -120,6 +125,17 @@ export default {
 
       return out;
     },
+
+    headers() {
+      const headers = this.$store.getters['type-map/headersFor'](this.schema, false);
+
+      if (this.paginationEnabled) {
+        // See https://github.com/rancher/dashboard/issues/10417, health comes from selectors applied locally to all pods (bad)
+        return headers.filter((h) => h.name !== WORKLOAD_HEALTH_SCALE.name);
+      }
+
+      return headers;
+    }
   },
 
   // All of the resources that we will load that we need for the loading indicator
@@ -129,6 +145,11 @@ export default {
 
   methods: {
     loadHeathResources() {
+      // See https://github.com/rancher/dashboard/issues/10417, health comes from selectors applied locally to all pods (bad)
+      if (this.paginationEnabled) {
+        return;
+      }
+
       // Fetch these in the background to populate workload health
       if ( this.allTypes ) {
         this.$fetchType(POD);
@@ -167,6 +188,7 @@ export default {
   <ResourceTable
     :loading="$fetchState.pending"
     :schema="schema"
+    :headers="headers"
     :rows="filteredRows"
     :overflow-y="true"
     :use-query-params-for-simple-filtering="useQueryParamsForSimpleFiltering"

diff --git a/shell/mixins/resource-fetch-api-pagination.js b/shell/mixins/resource-fetch-api-pagination.js
@@ -47,6 +47,7 @@ export default {
       const searchFilters = event.filter.searchQuery ? event.filter.searchFields.map((field) => new PaginationFilterField({
         field,
         value: event.filter.searchQuery,
+        exact: false,
       })) : [];
 
       const pagination = new PaginationArgs({