Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Application of selectors locally scales badly #10417

Open
richard-cox opened this issue Feb 7, 2024 · 7 comments
Open

Application of selectors locally scales badly #10417

richard-cox opened this issue Feb 7, 2024 · 7 comments

Comments

@richard-cox
Copy link
Member

richard-cox commented Feb 7, 2024

Blocked on rancher/rancher#46333

  • In places the UI fetches all of a resource and then filters by selector
    • see shell/utils/selector.js, in particular matches and matching
    • these are used in specific places but also findMatching if all resources have been fetched
  • Fetching all of a resource is bad and we need to fix this
  • Need to look at each use case and find a better way
    • in places we could just make an api request with labelSelector new filter=metadata.label instead
    • in other places individual resources each apply a selector to all resources (we couldn't just make a single request with a single selector)
  • Notable Parts
    • findMatching action
      • vue store action that will make an API request with the given selector
    • matching getter
      • vue store getter that will provide resources from cache that match the given selector
    • MatchExpressions
      • Component that creates a selector
    • ResourceSelector
      • Wraps MatchExpression, showing resources that match it's selector
    • selector.js helper functions
      • matches - does the given resource match the selector
      • matching - apply the given selector to the given resources
  • Specific cases
    • findMatching action (with selector)
      • shell/plugins/dashboard-store/resource-class.js
        • _findRelationship. selector from _relationshipsFor --> metadata.relationships[].selector
        • only used by shell/models/cis.cattle.io.clusterscan.js to find owned cluster scan CIS reports (cis.cattle.io.clusterscanreport)
          • Action: Remove generic handling
          • Action: Implement using API + selector
      • shell/pages/c/_cluster/longhorn/index.vue
        • services with label app=longhorn-ui
          • Action: Implement with API + label filtering
      • shell/pages/c/_cluster/uiplugins/CatalogList/CatalogUninstallDialog.vue
        • deployment, service, helm repo with specific label
          • Action: Implement with API + label filtering
      • shell/plugins/steve/subscribe.js
        • resyncing values (forced new request)
        • also uses matching getter --> selector.js matching
        • Action: Depends on clusterscan above
    • selector.js matches
      • shell/components/SortableTable/filtering.js handleStringSearch
        • converts sortable table search string to a selector, applies checks row
        • Action: None
      • shell/detail/workload/index.vue findMatchingServices
        • workload detail page. finds services associated with pods in a workload (service.spec.selector)
          • services shown on page
            • Action: Implement using API + selector
          • used to find ingresses (also shown on page)
            • Action: Implement using API + selector
      • shell/models/workload.js pods
        • find pods associated with workload (finds relationships that are pods, applies that selector to pods)
          • Action: Implement using API + selector
        • workload detail page
          • pod restarts
          • pod guages
          • job guages
          • services list
          • pods list
    • selector.js matching
      • shell/components/form/ResourceSelector.vue updateMatchingResources
        • ties in a MatchExpress component with a ResourceTable
        • only used by shell/edit/policy.poddisruptionbudget.vue
          • Action: Implement using API + selector
      • shell/edit/fleet.cattle.io.clustergroup.vue updateMatchingClusters
        • find clusters for cluster group mathing selector from MatchExpressions
          • Action: Implement using API + selector
      • shell/edit/service.vue updateMatchingPods
        • finds pods for the services matching service's value.spec.selector
          • Action: Implement using API + selector
      • shell/edit/networking.k8s.io.networkpolicy/index.vue
        • finds pods for policy matching selector from MatchExpressions
          • Action: Implement using API + selector
      • shell/edit/networking.k8s.io.networkpolicy/PolicyRuleTarget.vue getMatchingPods
        • find pods for rule matching selector from MatchExpressions
          • Action: Implement using API + selector
      • shell/edit/networking.k8s.io.networkpolicy/PolicyRuleTarget.vue getMatchingNamespaces
        • find namespaces for rule matching selector from MatchExpressions
          • Action: Implement using API + selector
      • shell/models/fleet.cattle.io.bundle.js targetClusters
        • find clusters for cluster group selector
          • Action: Implement using API + selector
        • find clusters for cluster selector
          • Action: Implement using API + selector
      • shell/models/fleet.cattle.io.clustergroup.js targetClusters
        • find clusters for cluster group selector
          • Action: Implement using API + selector
      • shell/models/fleet.cattle.io.gitrepo.js
        • find clusters for cluster group selector
          • Action: Implement using API + selector
        • find clusters for cluster selector
          • Action: Implement using API + selector
      • shell/models/service.js pods
        • find pods for service matching pod selector
          • Action: Implement using API + selector
      • shell/models/workload.js
        • find pods given workloads spec.selector (used for ssh to first running pod)
          • Action: Implement using API + selector

Useful links

Notes

Edit: Updates with all cases

@richard-cox richard-cox added this to the v2.9.0 milestone May 22, 2024
richard-cox added a commit to richard-cox/dashboard that referenced this issue May 28, 2024
richard-cox added a commit to richard-cox/dashboard that referenced this issue May 29, 2024
richard-cox added a commit that referenced this issue Jun 11, 2024
* Changes for new design
- New visuals
- Pagination controls --> load more
- finished testing of label select with pagination off

# Conflicts:
#	shell/edit/provisioning.cattle.io.cluster/__tests__/Basics.tests.ts

* Changes following review

* Update Node list to support server-side pagination
- Setup pagination headers for the node type
- Define a pattern for fetching custom list secondary resources
- Major improvements to the way pagination settings are defined and created
- Lots of docs improvements
- Handle calling fetch again once fetch is in progress (nuxt caches running request)
- Validate filter fields (not all are supported by the vai cache
- General pagination fixes

* Lint / test / fixes

* Improvements to configmap e2e test & Improve pagination disabled

* Beef up validation

* Fix missing name column in non-server-side paginated node list

* Fix PR automation actions
- fix syntax
- catch scenario where a pr has no fixed issue

> There's duplication between files, see #10534

* CI bump

* Fixes post merge

* Wire in 2.9.0 settings for server-side pagination
- Everything is gated on `on-disk-steve-cache` feature flag
  - There's a backend in progress item to resolve a `revision` issue, until then disable watching a resource given it
- Global Settings - Performance
  - Added new setting to enable server side pagination
  - this is incompatible with two other performance settings

* Integrate pagination with configmaps in cis clusterscanbenchmark edit form
Also
- improved labeled select pagination
- gate label select pagination functinality on steve cache being enabled

* - harvester machine-config
- project monitoring (and bug fixes)

* Disable workload screen if vai cache is on
- temp step until we get new overview

* TODOs and TEST

* Conditionally remove fetch of all secrets from SelectOrCreateAuthSecret

* TODOs and TEST

* Update SimpleSecretSelector
- only used in monitoring.coreos.com.alertmanagerconfig context

* View and Edit ingress - secrets

* node detail page - pods list

* Backup/Restore: Secrets (WIP)

* Backup/Restore: Secrets, and other usages of SimpleSecretSelector / SelectOrCreateAuthSecret

* Edit: Service account

* Add comments for remaining items

* Paginate Secret selection for logging providers
- Allow `None` option in Paginationed LabelSelect
- Optionally classify pagination response

* WIP

* fixes arfter merge

* Don't suggest container names, not practical
- previously all pods were fetched... and we scrapped all container names from them
- this is a scaling nightmare, user now must just enter the name/s to match

* Avoid findAll secrets in SimpleSecretSelector

* tidying up

* Move LabeledSelect/index.vue back to LabeledSelect.vue to not break extensions

* changes after self review... 1

* changes after self review... 2

* ooof

* changes after self review... 3

* fix formatting

* Link new paginated label select with pagination setting

* Work around failing kubewarden unit tests in check-plugins gate

* Fix backup.spec e2e test

* fix formatting, paginationUtils.isSteveCacheEnabled --> paginationUtils.isEnabled

* Don't fetch all secrets on cloud creds page

* Fix backup.spec e2e test

* TODO tidying / tracking

* don't getch ALL workloads for hacky way to get a link to a service's workload

* Fix bad merge

* Updates after working with vai cache image

* test fixes

* Create a convienence wrapper called ResourceLabelSelector that hides most of the complexity

* fix unit test

* Updates following review

* Remove workload health until #10417 is resolved

* Updates following review

* changes following self review

* Fix bottom bar of edit backup, edit restore pages

* revert temp change

* changes following self review

* Workaround for kubewarden unit tests in check plugin gate

* bump

* Fix e2e

* Fix linting

* type fixing

* - improve filtering without pagination
- update allowed fields given latest backend changes
- enable on by default exact filter string matches (disable for lists

* remove temp code

* fix linting

* Changes following review

* Fixes for vai cache feature flag
- name was changed from on-disk-steve-cache to ui-sql-cache
- fix timing issue - don't watch resources until we know the vai cache feature flag

* Changes following review

* Fix two sketchy tests
- new exception in docs page
- don't nav to page via button click and then goto same page

* More test improvement
- force user to go to tab which is source of route guard issue
- move setup stuff to a test for cypress to re-retry
richard-cox added a commit that referenced this issue Jun 11, 2024
…ginated (#11129)

* Changes for new design
- New visuals
- Pagination controls --> load more
- finished testing of label select with pagination off

# Conflicts:
#	shell/edit/provisioning.cattle.io.cluster/__tests__/Basics.tests.ts

* Changes following review

* Update Node list to support server-side pagination
- Setup pagination headers for the node type
- Define a pattern for fetching custom list secondary resources
- Major improvements to the way pagination settings are defined and created
- Lots of docs improvements
- Handle calling fetch again once fetch is in progress (nuxt caches running request)
- Validate filter fields (not all are supported by the vai cache
- General pagination fixes

* Lint / test / fixes

* Improvements to configmap e2e test & Improve pagination disabled

* Beef up validation

* Fix missing name column in non-server-side paginated node list

* Fix PR automation actions
- fix syntax
- catch scenario where a pr has no fixed issue

> There's duplication between files, see #10534

* CI bump

* Fixes post merge

* Wire in 2.9.0 settings for server-side pagination
- Everything is gated on `on-disk-steve-cache` feature flag
  - There's a backend in progress item to resolve a `revision` issue, until then disable watching a resource given it
- Global Settings - Performance
  - Added new setting to enable server side pagination
  - this is incompatible with two other performance settings

* Integrate pagination with configmaps in cis clusterscanbenchmark edit form
Also
- improved labeled select pagination
- gate label select pagination functinality on steve cache being enabled

* - harvester machine-config
- project monitoring (and bug fixes)

* Disable workload screen if vai cache is on
- temp step until we get new overview

* TODOs and TEST

* Conditionally remove fetch of all secrets from SelectOrCreateAuthSecret

* TODOs and TEST

* Update SimpleSecretSelector
- only used in monitoring.coreos.com.alertmanagerconfig context

* View and Edit ingress - secrets

* node detail page - pods list

* Backup/Restore: Secrets (WIP)

* Backup/Restore: Secrets, and other usages of SimpleSecretSelector / SelectOrCreateAuthSecret

* Edit: Service account

* Add comments for remaining items

* Paginate Secret selection for logging providers
- Allow `None` option in Paginationed LabelSelect
- Optionally classify pagination response

* WIP

* fixes arfter merge

* Don't suggest container names, not practical
- previously all pods were fetched... and we scrapped all container names from them
- this is a scaling nightmare, user now must just enter the name/s to match

* Avoid findAll secrets in SimpleSecretSelector

* tidying up

* Move LabeledSelect/index.vue back to LabeledSelect.vue to not break extensions

* changes after self review... 1

* changes after self review... 2

* ooof

* changes after self review... 3

* fix formatting

* Link new paginated label select with pagination setting

* Work around failing kubewarden unit tests in check-plugins gate

* Fix backup.spec e2e test

* fix formatting, paginationUtils.isSteveCacheEnabled --> paginationUtils.isEnabled

* Don't fetch all secrets on cloud creds page

* Fix backup.spec e2e test

* TODO tidying / tracking

* don't getch ALL workloads for hacky way to get a link to a service's workload

* Fix bad merge

* Updates after working with vai cache image

* test fixes

* Create a convienence wrapper called ResourceLabelSelector that hides most of the complexity

* fix unit test

* Add alt-loading indicator to sortable table, use for pagination

* Updates following review

* Remove workload health until #10417 is resolved

* Updates following review

* changes following self review

* Fix bottom bar of edit backup, edit restore pages

* revert temp change

* changes following self review

* Workaround for kubewarden unit tests in check plugin gate

* bump

* Fix e2e

* Fix linting

* type fixing

* fix standard loading indicator

* - improve filtering without pagination
- update allowed fields given latest backend changes
- enable on by default exact filter string matches (disable for lists

* remove temp code

* fix linting

* Changes following review

* Fixes for vai cache feature flag
- name was changed from on-disk-steve-cache to ui-sql-cache
- fix timing issue - don't watch resources until we know the vai cache feature flag

* Changes following review

* Fix two sketchy tests
- new exception in docs page
- don't nav to page via button click and then goto same page

* More test improvement
- force user to go to tab which is source of route guard issue
- move setup stuff to a test for cypress to re-retry

* changes following review
@richard-cox richard-cox modified the milestones: v2.9.0, v2.9.next1 Jun 17, 2024
@gaktive gaktive modified the milestones: v2.9.next1, v2.10.0 Jul 2, 2024
@nwmac nwmac modified the milestones: v2.10.0, v2.11.0 Jul 4, 2024
@richard-cox
Copy link
Member Author

Implementation dependent on rancher/rancher#46333

@richard-cox
Copy link
Member Author

richard-cox commented Jul 30, 2024

See rancher/rancher#46333 (comment)

More complex / full labelSelector support will not be included in 2.10 and are out of scope of the above linked issue.

For this release we will disable the basic use cases of selects x or y, to be enabled with full labelSelector support post 2.10 (issue pending).

If more core use cases are discovered then the above can be reviewed

Edit: More use cases have been discovered (description updated)

@richard-cox
Copy link
Member Author

@moio I've updated the issue including all the places where we apply fully selectors locally. There's quite a few.

There's a lot of work there to either

  • Disable all the features - I would not like to say we're production ready if so
  • Convert all usages to filter via API and not locally (requires full label selector api support)

@moio
Copy link
Contributor

moio commented Aug 19, 2024

@richard-cox: I agree this list looks too long to accept in a non-experimental version.

Disabling them while the feature is experimental does not make sense to me. Better to have a list of known regressions (or a shortlist of features expected to work) and actually address the underlying limitations.

I am adding full label selector API support to rancher/rancher#46333.

@richard-cox richard-cox modified the milestones: v2.10.0, v2.11.0 Sep 25, 2024
nwmac pushed a commit to nwmac/dashboard that referenced this issue Oct 24, 2024
* Changes for new design
- New visuals
- Pagination controls --> load more
- finished testing of label select with pagination off

# Conflicts:
#	shell/edit/provisioning.cattle.io.cluster/__tests__/Basics.tests.ts

* Changes following review

* Update Node list to support server-side pagination
- Setup pagination headers for the node type
- Define a pattern for fetching custom list secondary resources
- Major improvements to the way pagination settings are defined and created
- Lots of docs improvements
- Handle calling fetch again once fetch is in progress (nuxt caches running request)
- Validate filter fields (not all are supported by the vai cache
- General pagination fixes

* Lint / test / fixes

* Improvements to configmap e2e test & Improve pagination disabled

* Beef up validation

* Fix missing name column in non-server-side paginated node list

* Fix PR automation actions
- fix syntax
- catch scenario where a pr has no fixed issue

> There's duplication between files, see rancher#10534

* CI bump

* Fixes post merge

* Wire in 2.9.0 settings for server-side pagination
- Everything is gated on `on-disk-steve-cache` feature flag
  - There's a backend in progress item to resolve a `revision` issue, until then disable watching a resource given it
- Global Settings - Performance
  - Added new setting to enable server side pagination
  - this is incompatible with two other performance settings

* Integrate pagination with configmaps in cis clusterscanbenchmark edit form
Also
- improved labeled select pagination
- gate label select pagination functinality on steve cache being enabled

* - harvester machine-config
- project monitoring (and bug fixes)

* Disable workload screen if vai cache is on
- temp step until we get new overview

* TODOs and TEST

* Conditionally remove fetch of all secrets from SelectOrCreateAuthSecret

* TODOs and TEST

* Update SimpleSecretSelector
- only used in monitoring.coreos.com.alertmanagerconfig context

* View and Edit ingress - secrets

* node detail page - pods list

* Backup/Restore: Secrets (WIP)

* Backup/Restore: Secrets, and other usages of SimpleSecretSelector / SelectOrCreateAuthSecret

* Edit: Service account

* Add comments for remaining items

* Paginate Secret selection for logging providers
- Allow `None` option in Paginationed LabelSelect
- Optionally classify pagination response

* WIP

* fixes arfter merge

* Don't suggest container names, not practical
- previously all pods were fetched... and we scrapped all container names from them
- this is a scaling nightmare, user now must just enter the name/s to match

* Avoid findAll secrets in SimpleSecretSelector

* tidying up

* Move LabeledSelect/index.vue back to LabeledSelect.vue to not break extensions

* changes after self review... 1

* changes after self review... 2

* ooof

* changes after self review... 3

* fix formatting

* Link new paginated label select with pagination setting

* Work around failing kubewarden unit tests in check-plugins gate

* Fix backup.spec e2e test

* fix formatting, paginationUtils.isSteveCacheEnabled --> paginationUtils.isEnabled

* Don't fetch all secrets on cloud creds page

* Fix backup.spec e2e test

* TODO tidying / tracking

* don't getch ALL workloads for hacky way to get a link to a service's workload

* Fix bad merge

* Updates after working with vai cache image

* test fixes

* Create a convienence wrapper called ResourceLabelSelector that hides most of the complexity

* fix unit test

* Updates following review

* Remove workload health until rancher#10417 is resolved

* Updates following review

* changes following self review

* Fix bottom bar of edit backup, edit restore pages

* revert temp change

* changes following self review

* Workaround for kubewarden unit tests in check plugin gate

* bump

* Fix e2e

* Fix linting

* type fixing

* - improve filtering without pagination
- update allowed fields given latest backend changes
- enable on by default exact filter string matches (disable for lists

* remove temp code

* fix linting

* Changes following review

* Fixes for vai cache feature flag
- name was changed from on-disk-steve-cache to ui-sql-cache
- fix timing issue - don't watch resources until we know the vai cache feature flag

* Changes following review

* Fix two sketchy tests
- new exception in docs page
- don't nav to page via button click and then goto same page

* More test improvement
- force user to go to tab which is source of route guard issue
- move setup stuff to a test for cypress to re-retry
nwmac pushed a commit to nwmac/dashboard that referenced this issue Oct 24, 2024
…ginated (rancher#11129)

* Changes for new design
- New visuals
- Pagination controls --> load more
- finished testing of label select with pagination off

# Conflicts:
#	shell/edit/provisioning.cattle.io.cluster/__tests__/Basics.tests.ts

* Changes following review

* Update Node list to support server-side pagination
- Setup pagination headers for the node type
- Define a pattern for fetching custom list secondary resources
- Major improvements to the way pagination settings are defined and created
- Lots of docs improvements
- Handle calling fetch again once fetch is in progress (nuxt caches running request)
- Validate filter fields (not all are supported by the vai cache
- General pagination fixes

* Lint / test / fixes

* Improvements to configmap e2e test & Improve pagination disabled

* Beef up validation

* Fix missing name column in non-server-side paginated node list

* Fix PR automation actions
- fix syntax
- catch scenario where a pr has no fixed issue

> There's duplication between files, see rancher#10534

* CI bump

* Fixes post merge

* Wire in 2.9.0 settings for server-side pagination
- Everything is gated on `on-disk-steve-cache` feature flag
  - There's a backend in progress item to resolve a `revision` issue, until then disable watching a resource given it
- Global Settings - Performance
  - Added new setting to enable server side pagination
  - this is incompatible with two other performance settings

* Integrate pagination with configmaps in cis clusterscanbenchmark edit form
Also
- improved labeled select pagination
- gate label select pagination functinality on steve cache being enabled

* - harvester machine-config
- project monitoring (and bug fixes)

* Disable workload screen if vai cache is on
- temp step until we get new overview

* TODOs and TEST

* Conditionally remove fetch of all secrets from SelectOrCreateAuthSecret

* TODOs and TEST

* Update SimpleSecretSelector
- only used in monitoring.coreos.com.alertmanagerconfig context

* View and Edit ingress - secrets

* node detail page - pods list

* Backup/Restore: Secrets (WIP)

* Backup/Restore: Secrets, and other usages of SimpleSecretSelector / SelectOrCreateAuthSecret

* Edit: Service account

* Add comments for remaining items

* Paginate Secret selection for logging providers
- Allow `None` option in Paginationed LabelSelect
- Optionally classify pagination response

* WIP

* fixes arfter merge

* Don't suggest container names, not practical
- previously all pods were fetched... and we scrapped all container names from them
- this is a scaling nightmare, user now must just enter the name/s to match

* Avoid findAll secrets in SimpleSecretSelector

* tidying up

* Move LabeledSelect/index.vue back to LabeledSelect.vue to not break extensions

* changes after self review... 1

* changes after self review... 2

* ooof

* changes after self review... 3

* fix formatting

* Link new paginated label select with pagination setting

* Work around failing kubewarden unit tests in check-plugins gate

* Fix backup.spec e2e test

* fix formatting, paginationUtils.isSteveCacheEnabled --> paginationUtils.isEnabled

* Don't fetch all secrets on cloud creds page

* Fix backup.spec e2e test

* TODO tidying / tracking

* don't getch ALL workloads for hacky way to get a link to a service's workload

* Fix bad merge

* Updates after working with vai cache image

* test fixes

* Create a convienence wrapper called ResourceLabelSelector that hides most of the complexity

* fix unit test

* Add alt-loading indicator to sortable table, use for pagination

* Updates following review

* Remove workload health until rancher#10417 is resolved

* Updates following review

* changes following self review

* Fix bottom bar of edit backup, edit restore pages

* revert temp change

* changes following self review

* Workaround for kubewarden unit tests in check plugin gate

* bump

* Fix e2e

* Fix linting

* type fixing

* fix standard loading indicator

* - improve filtering without pagination
- update allowed fields given latest backend changes
- enable on by default exact filter string matches (disable for lists

* remove temp code

* fix linting

* Changes following review

* Fixes for vai cache feature flag
- name was changed from on-disk-steve-cache to ui-sql-cache
- fix timing issue - don't watch resources until we know the vai cache feature flag

* Changes following review

* Fix two sketchy tests
- new exception in docs page
- don't nav to page via button click and then goto same page

* More test improvement
- force user to go to tab which is source of route guard issue
- move setup stuff to a test for cypress to re-retry

* changes following review
@richard-cox
Copy link
Member Author

We should be able to get back workload health indicators once this work is complete (see shell/list/workload.vue loadHeathResources)

@richard-cox
Copy link
Member Author

Note - some affected places in code now reference this issue. ensure those are addressed when resolving this

@richard-cox
Copy link
Member Author

Blocked on rancher/rancher#46333

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants