Can't list Apps or Workloads through UI in projects with many resources

[fabiorauber]

What kind of request is this (question/bug/enhancement/feature request):
Bug

Steps to reproduce (least amount of steps as possible):
Create about 800 Apps in a single Project, with more than 1 pod per app and make updates to each app so that the total of ConfigMaps is over 1600. Each Pod may have more than one volume, so the total of PVCs is over 2500. Try to display the Workloads section of the Project or the Apps section.

Result:

Get https://172.31.21.245:6443/api/v1/namespaces/sapl-xdu44/services?timeout=30s: context deadline exceeded

Rancher UI goes back to list the clusters, and do not display the desired Apps/Workloads section.

Other details that may be helpful:

When this happens, load of the kube-apiserver spikes up to use all processors (in this case, 8) of the controller and etcd logs several errors like this:

etcdserver: read-only range request "key:\"/registry/serviceaccounts/cattle-system/kontainer-engine\" " with result "range_response_count:1 size:255" took too long (298.036652ms) to execute

In addition, kube-apiserver logs many errors like the following:

E0717 14:19:35.239923       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}
E0717 14:19:35.261172       1 writers.go:168] apiserver was unable to write a JSON response: http: Handler timeout
E0717 14:19:35.261487       1 status.go:64] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}

Environment information

• Rancher version: 2.2.5
• Installation option (single install/HA): single install.

Cluster information

• Cluster type: Hosted XenServer Docker machine driver
• Machine type (cloud/VM/metal) and specifications (CPU/memory): All XenServer VMs, running Ubuntu 18.04 LTS. Controllers/ETCD servers (3 for the cluster) and the single Rancher server have all 8 CPUs and 10GB of RAM.
• Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

• Docker version (use docker version):

Docker version 18.09.2, build 6247962

gzrancher/rancher#10647

[fabiorauber]

Seems to have relation with #16605. But in my case, there is no excessive ConfigMaps to delete.

[albertmatyi]

I've just updated to 2.2.6 and the UI became responsive. We've been on 2.2.1 and sometimes the UI came to a halt, couldn't access projects, or was really slow.

[fabiorauber]

The error persists on version 2.2.6, at least for my clusters.

[fabiorauber]

Still happening in version 2.2.7

[fabiorauber]

Still happening in version 2.2.8

[fabiorauber]

Tested on 2.3.5, Kubernetes version 1.15.11.

Error
Get https://172.31.20.52:6443/api/v1/namespaces/portal-baiao-pa/pods?timeout=30s: context deadline exceeded

[fabiorauber]

This other cluster has 800 Apps in a single project. As you can see, this time the UI responded, but barely passed the 30 seconds limit:

This test was made on a Kubernetes 1.17.4 cluster with three Control Plane + ETCD nodes and 7 worker nodes, Etcd version 3.4.3, on Rancher v2.3.5.

[fabiorauber]

Similar to #18522, but with ConfigMaps