[BUG] Host not found in upstream when using bridge network with nginx

[xave]

Description

A host not found in upstream error is thrown by nginx when trying to use the name of a service when setting up nginx reverse proxy. The service is built on a different machine than the server and its image is synced. That service has been verified to work as intended on its own on the server machine (before the tls setup). Both the service (web app) and nginx are connected to a named bridge network my_bridge_network. The service is run with the flag to use that network. The nginx instance has this network in its docker-compose service. The expectation is that in the nginx.conf file one can reference the name of the service. But the nginx runtime error demonstrates otherwise.

Steps To Reproduce

1. Two docker compose files and a bridge network. The first one creates an app attached to a bridge network called my_bridge_network. The image from this is synced to a server.
2. The second docker compose is run on the server to configure nginx and ssl. It has a network directive with the same network (spelled the same way).
3. The synced image is run with docker run --network my_bridge_network app_image_name
4. In nginx.conf, I should be able to create a block with location / { proxy_pass http://app} because the image for the app is made from a docker compose service titled app:. Docker creates a container called my-directory-app.
5. nginx does not recognize this: host not found in upstream "app" in /etc/nginx/conf.d/default.conf:<lineNo>

Compose Version

v2.32.4

Docker Environment

Anything else?

No response

[ndeloof]

I'm not sure I get it right. Does this error occur in the container ran by compose, or in the second one you run by docker run ...

I should be able to create a block with location / { proxy_pass http://app} because the image for the app is made from a docker compose service titled app:

The host name is not part of a docker image, this is a runtime information.

[xave]

Thank you. Let me elaborate.

The typical path would be to set everything up in one single docker compose file. So one might expect something like (abbreviated)

services:
    nginx:
        ...
        networks:
            - nginx_proxy_network
    my-fantastic-app:
        ...
        networks:
            - nginx_proxy_network
 networks:
    nginx_proxy_network:
        external: true

Then in the nginx config, you could access it as above in the nginx.conf proxy_pass declaration. This is an established pattern all over the web that people do.

My case is different. I have two docker compose setups. One sets up nginx only with certbot and then has that configured nginx listening. Then I have another image made with another docker compose file that I want to reverse proxy into this configured nginx.

It seems I need either the IP address of that process or to be able to do the location / { proxy_pass http://my-fantastic-app}/ trick. But that does not work for me.

In my setup, I still have the my-fantastic-app and the nginx using the nginx_proxy_network. I call docker run --network nginx_proxy_network my_app_image on the app's image. I want to be able to reference that in the nginx file the way you could do with a single docker compose file setup.

Please let me know if you need additional clarity on my setup.

[xave]

To clarify, the error is a runtime error by nginx when it tries to access that route.

nginx: [emerg] host not found in upstream "my-fantastic-app"  in /etc/nginx/conf.d/default.conf

But the solution I am using to try and get it to work involves Docker fundamentally.

[xave]

My thought is that Docker does something for you automatically when you have the single compose file and that I have to manually recreate that with this setup.

[ndeloof]

Compose "just" creates your application container with my-fantastic-app alias on network, so that other containers connected to this network. You can check using docker inspect:

            "Networks": {
                "test_nginx_proxy_network": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "test-my-fantastic-app-1",
                        "my-fantastic-app"
                    ],

[xave]

I just ran docker inspect on my network and some key things are different. I don't have the following fields:

• Aliases
• Links
• IPAMConfig (that is separated into "IPAM": with a "Config": child)

The output does show me IP addresses, Subnet and Gateway. I am pretty sure these are dynamic, though.

When I query my app docker inspect my_app_image, I get a field:

    "Config":
        "Labels":
            "com.docker.compose.service": app

My understanding is this service name is what goes into the nginx.conf. Since my service is called app, then I should be able to

location / { 
    proxy_pass http://app
    ...
    }

Since this does not work, there is an extra step it seems. I am trying to figure that out here with this issue. It would work if I had app as a service in the docker compose file with nginx. But that is not my setup for other reasons.

So the open question is still with this particular workflow, what is the best way to get that information to nginx in an automated way for my specific image?

Would I query my network and update the nginx.conf dynamically?

[ndeloof]

You must inspect the running container. Image doesn't store any of the network settings, those a pure runtime configuration.

[xave]

I run the image with the network. I am expecting that all things on the network are aware of that.

So is this not the correct behavior:

• docker run -dit --network nginx_proxy_network my_app_image
• docker compose up (this is the nginx container, which loads an nginx.conf file on startup.
• The nginx.conf file has access to the service name of the app image (app).

You do get this behavior with a single compose with everything in it.

[xave]

To clarify, this is an example of a working nginx.conf solution for a compose file including a service called backend and one calling nginx:

docker-compose.yaml:

services:
  nginx:
    image: my-nginx
    ports:
      - "80:80"
    networks:
      - nginx_proxy_network
  backend:
    image: my-backend
    networks:
      - nginx_proxy_network
networks:
  nginx_proxy_network:
    external: true

nginx.conf:

http {
    server {
        listen 80;
        server_name example.com;
        return 301 https://$server_name$request_uri;
    }

    server {
        listen 443 ssl;
        server_name example.com;

        ssl_certificate /etc/nginx/certs/example.com.crt;
        ssl_certificate_key /etc/nginx/certs/example.com.key;

        location / {
            proxy_pass http://backend:8080;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }
    }
}

The line proxy_pass http://backend:8080; refers to the name backend exposed by Docker itself. It is exactly the service name in the docker-compose file (i.e. if the spelling of one changes, so must the other to match).

I want to do the same thing with my service called app. But this example (above) has everything declared in a single compose file. I have one file (below) for nginx setup and another image for the app. So I have:

My docker-compose.yaml:

services:
  nginx:
    image: my-nginx
    ports:
      - "80:80"
    networks:
      - nginx_proxy_network
networks:
  nginx_proxy_network:
    external: true

As of now, if I try to do the same setup but with a separate image instead of an app service in the docker-compose, nginx cannot find app on the proxy_pass line. In the above example it can find the respective service backend. So something is slightly different that I need to address. What happens automatically above does not happen in my case.

[xave]

You must inspect the running container. Image doesn't store any of the network settings, those a pure runtime configuration.

@ndeloof , I think I understand now. Please let me know if this is what you meant.

I just ran docker inspect nginx_proxy_network and it showed my_app_image as being attached to it. It did NOT show my nginx from the docker compose attached to it.

So I have been expecting that the docker compose up with the network options would preserve that connection, but it does not.

I ran docker network connect nginx_reverse_proxy my-nginx and verified that it lists as connected. The challenge is still that the intent is for the docker-compose.yaml to configure nginx and have it ready to wait for an app to reverse proxy. But this nginx workflow will still fail on compose if I have to add the nginx to the network after the docker compose up.

I had hoped I could spin up the app first, run the docker compose up second (so the app exists for the nginx.conf DNS resolution), and that everything would work together from that point.

[xave]

I believe I need to use the Docker DNS resolver 127.0.0.11 to access the nginx_reverse_proxy network in the nginx.conf. I will experiment.

[xave]

That did it.

Something like

upstream foobar {
    resolver 127.0.0.11;
    server app:80;
}

server {
    listen 80;
    listen 443;
    location / {
        proxy_pass http://foobar;
        ...
    }

where foobar is an arbitrary identifier you want to use in the nginx.conf and app is the name of the running container.