[BUG] Generally buggy behaviour when proxying Nextcloud

[CorneliusCornbread]

So I'm attempting to proxy my nextcloud instance, I've got the trusted IPs setup in nextcloud and have had this nextcloud instance proxied successfully via NPM in the past. However, it seems like Nextcloud really doesn't like being proxied via Zoraxy. Looking at the browser console, it seems like it's not properly proxying resource requests to be done via the domain, rather trying to directly access the resources via either the docker IP or the local IP address.

To Reproduce
Nextcloud proxy:

Changing to the local IP address instead of the docker one will simply result in it working, but only VIA LAN connections.

Expected behavior
Nextcloud should load as normal.

Browser (if it is a bug appears on the UI section of the system):

• OS: Fedora (tested non LAN connections via phone data on Android)
• Browser: Firefox
• Version: 126.0

Host Environment (please complete the following information):

• Arch: x86
• Device: Ryzen 3600x system
• OS: Unraid
• Version: 6.12.10

Additional context
I cannot reproduce this behaviour with my Crafty or Dozzle instance, they work just fine.

[tobychui]

I guess either your network setting have issues or your NextCloud setting is wrong. I don't use NextCloud, but this similar issue sometime happens to Wordpress too, which was caused by invalid settings of based URL / domains.
As you can see from the first part of the error message, your next cloud web interface trying to load the resources (svg) from its docker IP address

But from the later part, I assume you are connecting to it via local IP address, due to self src requirement, it cannot load as both address (docker IP start with 172 and your connecting IP start with 192) doesn't match.

This is a common issue with apps that didn't design to be reverse proxied. So to make it correctly load, you should be able to set the base URL / domain like wordpress that point to your (sub)domain name and force NextCloud to load resources from your (sub)domain URL instead of 172.

[Morethanevil]

If you are using Nextcloud in docker, use the normal HTTP Port (80). SSL is done via Zoraxy, uncheck "Require TLS" and "Skip verififcation" and uncheck "Allow Plain HTTP"

Require TLS and Allow plain HTTP are exactly the opposite :)

I have Nextcloud running on my host. Zoraxy points to my Apache webserver which is serving Nextcloud. It works on the normal HTTP Port. Certificates are handled by Zoraxy, you should avoid additional HTTPS ports or configuration.

Do you use the AIO image from Nextcloud?

[nettybun]

Nextcloud is getting confused because Zoraxy is sending the HTTP Host header of the endpoint, not the domain. If you configure Zoraxy to route "nextcloud.example.com" to "127.0.0.1:8443" you'll get HTTP Host header as 127.0.0.1 and HTTP Referrer header as https://127.0.0.1:8443

I'm not sure if this is by design, but Nextcloud doesn't like it.

The image I have is lscr.io/linuxserver/nextcloud:latest which uses nginx+php-fpm. I'm also running Podman rootless as the container engine.

Here are the nginx error.log and access.log:

root@c8611a770e26:/config/log/nginx# tail error.log 
2024/05/23 00:43:46 [error] 392#392: *3 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.88.0.11, server: _, request: "POST / HTTP/2.0", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1:8443"
root@c8611a770e26:/config/log/nginx# tail access.log 
10.88.0.13 - - [23/May/2024:01:00:55 -0700] "GET / HTTP/2.0" 302 0 "-" "Go-http-client/2.0"
10.88.0.13 - - [23/May/2024:01:00:55 -0700] "GET /index.php/login HTTP/2.0" 200 14824 "https://127.0.0.1:8443" "Go-http-client/2.0"
10.88.0.13 - - [23/May/2024:01:05:55 -0700] "GET / HTTP/2.0" 302 0 "-" "Go-http-client/2.0"
10.88.0.13 - - [23/May/2024:01:05:55 -0700] "GET /index.php/login HTTP/2.0" 200 14832 "https://127.0.0.1:8443" "Go-http-client/2.0"

The Go-http-client is Zoraxy.

The container image I'm using has this Docker-specific line:

    # display real ip in nginx logs when connected through reverse proxy via docker network
    set_real_ip_from 172.16.0.0/12;
    real_ip_header X-Forwarded-For;

I changed that CIDR from Docker's 172.x to Podman's 10.88.0.0/16 and then suddenly my logs were fixed:

root@c8611a770e26:/config/log/nginx# tail access.log 
107.11.32.102 - - [23/May/2024:01:55:12 -0700] "GET / HTTP/2.0" 302 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"
107.11.32.102 - - [23/May/2024:01:55:13 -0700] "GET /index.php/login HTTP/2.0" 200 14832 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"

Still, the buggy broken links remain in the UI.

I think it's this fastcgi param line that is giving PHP the wrong server name?

root@c8611a770e26:/# nginx -T | grep fastcgi_param
...
fastcgi_param  SERVER_NAME        $host; # Send HTTP_HOST as SERVER_NAME. If HTTP_HOST is blank, send the value of server_name from nginx (default is `_`)

I asked nginx to log its $http_host and $host to confirm Zoraxy is sending the wrong header:

    # Sets the path, format, and configuration for a buffered log write.
    log_format hostcombined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    'HOST:"$host" HTTP_HOST:"$http_host" REF:"$http_referer" "$http_user_agent"';
    access_log /config/log/nginx/access.log hostcombined;

It seems:

107.11.32.102 - - [23/May/2024:02:17:08 -0700] "PUT /ocs/v2.php/apps/user_status/api/v1/heartbeat?format=json HTTP/2.0" 200 149 HOST:"127.0.0.1" HTTP_HOST:"127.0.0.1:8443" REF:"-" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"
107.11.32.102 - - [23/May/2024:02:17:14 -0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0" 304 0 HOST:"127.0.0.1" HTTP_HOST:"127.0.0.1:8443" REF:"-" "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0"

So to workaround this in Nextcloud (or any downstream app that relies on HTTP Host header) you'll have to hardcode the nginx/apache server config to your domain... (note: nginx does not let you modify $host!)

fastcgi_param SERVER_NAME nextcloud.example.com

Fixed it for me! 👍

[tobychui]

@nettybun Thanks for the details debug and fixing process!
I will try to test this out later this weekend and see if I need to introduce a new switch for automatic hostname rewriting based on inbound domain name.

[tobychui]

That is interesting, as I am testing Zoraxy natively on my system (Windows) with a debug php script, I get the followings.

<?php
$hostname = $_SERVER['HTTP_HOST'];
echo $hostname;
echo '<br>';

$hostname = $_SERVER['SERVER_NAME'];
echo $hostname;
echo '<br>';

$serverIp = $_SERVER['SERVER_ADDR'];
echo $serverIp;
echo '<br>';

?>

Results

test.imuslab.internal
test.imuslab.internal
127.0.0.1

So php indeed grab the correct HTTP_HOST sent from Zoraxy. Besides, the Go-http-client should not be coming from Zoraxy, as in Zoraxy source code, the user agent information was removed from the outgoing request to the back-end server in dpcore.go

//Hide Go-HTTP-Client UA if the client didnt sent us one
	if _, ok := header["User-Agent"]; !ok {
		// If the outbound request doesn't have a User-Agent header set,
		// don't send the default Go HTTP client User-Agent.
		header.Set("User-Agent", "")
	}

My assumption is that there is something wrong between Zoraxy and the Nginx server running NextCloud (php) and I cannot reproduce it with my environment. If anyone is available, please help by use the attached debug.php file and provide more information regarding setups that have shows this bug.
debug.zip

Updates

Ok, I can reproduce this issue if and only if I run Zoraxy in docker and using the docker image from linuxserver/nextcloud. Running Zoraxy in docker and proxy to a php server will shows the correct HTTP_HOST and running Zoraxy natively and NextCloud in docker will also show correct UI.

Here is my portainer / container setup with mydomain.com -> 192.168.1.202:8443 (NextCloud instance, Zoraxy proxy rule is set to TLS enabled and skip TLS validation check)

Here is the result when Zoraxy is running natively (no docker) and NextCloud in container

Here is result when Zoraxy is running in docker and connecting to a PHP server, the HTTP_HOST header is correctly shown.

Updates 2

Try to comment this line (in the actual default.conf file in container volume) and see if everything just magically works again?

https://github.com/linuxserver/docker-nextcloud/blob/dafdd3d0efa43c9527c7d4d1fec3bbf4f8e2a610/root/defaults/nginx/site-confs/default.conf.sample#L24

[nettybun]

@tobychui whoa! Thanks for putting so much effort into it. However, I worry you may be seeing the same issue as me:

If you hover on these thumbnails it will say http://192.168...? Maybe? Those thumbnails for me were pointing to http://127.0.0.1:8443

I'll be back at a laptop soon and I can test with your debug script.

[nettybun]

I used your debug.php script by directly putting it into the nextcloud image under /app/www/public (but nextcloud disallows unknown php files since that's dangerous? I think? so I had to re-use/modify an existing file; I chose status.php)

Seems like the HTTP Host header is mangled somehow very odd... I'll keep investigating!

[nettybun]

ooo FOUND SOMETHING.

I think Zoraxy has different Go HTTP Clients for HTTP vs HTTPS? You're right that HTTP is OK, but Nextcloud uses HTTPS! I spin up two https://hub.docker.com/r/traefik/whoami containers (no PHP, written in Go):

$ podman run --name echo-http --detach --rm --replace -p 2000:2000 --replace docker.io/traefik/whoami:latest -name echo-http-server --port 2000
$ mkdir ./tls-test
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./tls-test/key.pem -out ./tls-test/cert.pem
$ podman run --name echo-https --detach --rm -v ./tls-test/:/tls/ -p 2443:2443 docker.io/traefik/whoami:latest -name echo-https-server -port 2443 -cert /tls/cert.pem -key /tls/key.pem

Note that echo. is HTTP and dns. is HTTPS:

The host headers are different! Zoraxy is sending the correct one for HTTP, but incorrect for HTTPS.

I believe this IS a bug in Zoraxy. Hope this is helpful!

[nettybun]

This looks like the cause of the bug. @tobychui I don't understand the code comment; can you advise if it is safe to modify this?

I modify the code:

Rebuild with go build.

Yes this is the issue:

I modify the code again:

Problem fixed!

This also fixed Nextcloud! See how now the thumbnails are working - they aren't "?" anymore, they are the real thing:

Can we push this fix?

I worry the code comment is telling me the change is not this easy though >_>

[tobychui]

After adding the domain in the config.php inside the docker volume and patched PR #168, this bug has been fixed.

I guess this bug also partly fixed the Proxmox issues which I will investigate later. Thanks so much for everyone involved and help making Zoraxy a better tool!