Uploading files over 1Mb fails with 413 file too large

[MaximUltimatum]

Describe your Issue

When attempting to upload files over 1M, the client (desktop windows, desktop linux, android, or directly from web GUI) fails with 413 file too large.
I re-enabled debug logging in my helm chart. However, I wasn't seeing the upload failures correspond to any log messages in the nextcloud container.
However, when I checked the ingress nginx logs, I am seeing the below error

Logs and Errors

From the ingress-nginx logs

2023/11/28 17:53:58 [error] 27#27: *883 client intended to send too large body: 8928872 bytes, client: 10.233.64.0, server: fun-domain-name.org, request: "PUT /remote.php/webdav/testing_shared/A_fun_8Mb_file.pdf HTTP/2.0", host: "fun-domain-name.org"

I have checked the corresponding timeframe with debug logging on in the nextcloud container - it registers that there was a GET request for the page when I loaded it, but nothing about the failure to upload the file.

Attempts to solve the issue I've made so far

I've read through this post on the nextcloud forums and [https://github.com/nextcloud/docker/issues/762#issuecomment-504225433] this github issue which appear to be very similar symptoms.

I think my biggest problem is I'm not sure how to properly configure the accepted client size on the ingress-nginx (reverse-proxy?) ingress controller. I'm a little shaky on the interaction between the ingress nginx stuff that helm sets up and the ingress-controller, that (I believe?) I configured on my own. For avoidance of confusion, I'll post the ingress setup directly instead of trying to describe it and risk mistating something.

kubectl get ingress -n default
NAME               CLASS   HOSTS                   ADDRESS         PORTS     AGE
nextcloud-friday   nginx   a-fun-domain.org   10.233.28.200   80, 443   17d

Name:             nextcloud-friday
Labels:           app.kubernetes.io/component=app
                  app.kubernetes.io/instance=nextcloud-friday
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=nextcloud
                  helm.sh/chart=nextcloud-4.5.0
Namespace:        default
Address:          10.233.28.200
Ingress Class:    nginx
Default backend:  <default>
TLS:
  fun-domain-name-cert terminates a-fun-domain-name.org
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  fun-domain-name.org  
                         /   nextcloud-friday:8080 (10.233.67.65:80)
Annotations:             cert-manager.io/cluster-issuer: letsencrypt-prod
                         kubernetes.io/tls-acme: true
                         meta.helm.sh/release-name: nextcloud-friday
                         meta.helm.sh/release-namespace: default
                         nginx.ingress.kubernetes.io/server-snippet:
                           # server_tokens off;
                           # proxy_hide_header X-Powered-By;
                           # rewrite ^/.well-known/webfinger /index.php/.well-known/webfinger last;
                           # rewrite ^/.well-known/nodeinfo /index.php/.well-known/nodeinfo last;
                           # rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
                           # rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json;
                           location = /.well-known/carddav {
                             return 301 $scheme://$host/remote.php/dav;
                           }
                           location = /.well-known/caldav {
                             return 301 $scheme://$host/remote.php/dav;
                           }
                           client_max_body_size 10G;
                           client_body_buffer_size 400M;
                           # location = /robots.txt {
                           #   allow all;
                           #   log_not_found off;
                           #   access_log off;
                           # }
                           # location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
                           #   deny all;
                           # }
                           # location ~ ^/(?:autotest|occ|issue|indie|db_|console) {
                           #   deny all;
                           # }
Events:                  <none>



kubectl get deployments -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           39d

kubectl describe deployment -n ingress-nginx ingress-nginx-controller
Name:                   ingress-nginx-controller
Namespace:              ingress-nginx
CreationTimestamp:      Fri, 20 Oct 2023 16:27:23 -0500
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=ingress-nginx
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=ingress-nginx
                        app.kubernetes.io/part-of=ingress-nginx
                        app.kubernetes.io/version=1.9.4
                        helm.sh/chart=ingress-nginx-4.8.3
Annotations:            deployment.kubernetes.io/revision: 2
                        meta.helm.sh/release-name: ingress-nginx
                        meta.helm.sh/release-namespace: ingress-nginx
Selector:               app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/managed-by=Helm
                    app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    app.kubernetes.io/version=1.9.4
                    helm.sh/chart=ingress-nginx-4.8.3
  Service Account:  ingress-nginx
  Containers:
   controller:
    Image:       registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:some-fun-sha-chars
    Ports:       80/TCP, 443/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
  Volumes:
   webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Progressing    True    NewReplicaSetAvailable
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ingress-nginx-controller-cd7b768cc (1/1 replicas created)
Events:          <none>

What I suspect is happening

Typing this all out, I'm wondering if, since the logs don't show up in the nextcloud-pod log and the logs do show up in the pod managed by the nginx-controller-deployment I've incorrectly set up my ingress-controller.
I've tried exec-ing into the ingress-controller container directly and adding the client_max_body_size 10G; and client_body_buffer_size 400M; directly into the nginx.conf file. That didn't work, but I'm wondering if my attempts to restart nginx after editing the file didn't actually restart it.
If this is the issue, I don't think it's necessarily related to the nextcloud-helm chart (and more my own lack of understanding of ingress controllers).

Describe your Environment

• Kubernetes distribution: Vanilla k8s deployed via kubespray

• Helm Version (or App that manages helm):

helm version
version.BuildInfo{Version:"v3.13.2", GitCommit:"2a2fb3b98829f1e0be6fb18af2f6599e0f4e8243", GitTreeState:"clean", GoVersion:"go1.20.10"}

• Helm Chart Version:

• values.yaml: I have a testing branch with current values file here

Additional context, if any

Apologies for submitting another issue so soon. I don't know if I'm off the happy path or just good at finding/creating problems :).
Thank you for your help in advance!

[MaximUltimatum]

On the chance that google has lead you here and you are facing a similar issue.... don't comment out lines in a helm chart when debugging something... and then forget to uncomment them.
https://github.com/MaximUltimatum/kube-homelab/blob/master/helm-configs/nextcloud-values.yaml#L28
This was the problem nginx.ingress.kubernetes.io/proxy-body-size: 4G was commented out

[jessebot]

This was the problem nginx.ingress.kubernetes.io/proxy-body-size: 4G was commented out

Yep, that's how I solved it too a while back. Good sleuthing!