nginx fails to start on OKD 3.11

[ghost]

ISSUE TYPE

• Bug Report

SUMMARY

When AWX is deployed on OKD, it seems like its nginx web server can't start due to permissions denied error on port 80.
The StatefulSet-dedicated pod is still marked as running, yet you cannot reach the login mask through the route as OpenShift returns the "Application not available" error.
I attached the log of awx-web container where you can see the error.

ENVIRONMENT

• AWX version: 9.0.0
• AWX install method: openshift 3.11
• Ansible version: 2.8.5
• Operating System: CentOS 7.7
• Web Browser: Firefox 70.0

STEPS TO REPRODUCE

1. Install AWX 9.0.0 on OKD 3.11
2. Try to reach the WebUI via route.

EXPECTED RESULTS

Reach the login screen.

ACTUAL RESULTS

OKD returns the error "Application is not available".

ADDITIONAL INFORMATION

awx-0.log

[ingvaldlorentzen]

I'm experiencing the same issue with AWX 9.0.0 in OpenShift 3.11.16

Was not an issue with AWX 8.0.0 and 7.0.0 in the exact same environment.

ENVIRONMENT

• AWX version: 9.0.0
• AWX install method: OpenShift manual deployment

- [warn] 158#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:5
- INFO exited: nginx (exit status 1; not expected)
- [emerg] 158#0: bind() to 0.0.0.0:80 failed (13: Permission denied)

[ryanpetrello]

@shanemcd do you think this is related to our CentOS 8 move or @wenottingham's recent nginx config change?

[ffirg]

Snap, same issue when using CRC
$ crc version
crc version: 1.0.0+575079b
OpenShift version: 4.2.0 (embedded in binary)

[ffirg]

This is an /etc/nginx/nginx.conf problem.

8.0.0 has the default server using port 8052
9.0.0 has port 80 hence the permission denied running as awx

[nkiraly]

Best I can tell, this change
712b07c#diff-a2ef2d6347894a92a6b882e3793fb06c

has resulted in published awx_web 9.0.0 image having the default nginx.conf instead of the awx repository's nginx.conf.

[nkiraly]

AWX web gets served as expected for me in my kubernetes cluster if take the awx_web:9.0.0 release image and copy in an nginx.conf based on the template installer/roles/local_docker/templates/nginx.conf.j2 which was renamed from installer/roles/image_build/templates/nginx.conf.j2 transitioning from release 8.0.0 to 9.0.0

[dmaple-gci]

Best I can tell, this change
712b07c#diff-a2ef2d6347894a92a6b882e3793fb06c

has resulted in published awx_web 9.0.0 image having the default nginx.conf instead of the awx repository's nginx.conf.

Yes, I agree. The nginx.conf was taken out of the image_build role and only put in the local_docker and docker-compose. This leaves image build for k8s and OKD with default nginx.conf and an unusable awx_web container.

[shanemcd]

Oops, apologies for not catching this. Going to work on a patch today or tomorrow.

[shanemcd]

This should be fixed in 9.0.1.

[ghost]

Yes, I can confirm that now it's working properly.

Thanks @ryanpetrello and @shanemcd for your support!

[zen4ever]

I'm experiencing the same issue with an image ansible/awx_web:9.0.1 from DockerHub

[jmereaux]

Same issue with version 9.1.1 of the image. Wasn’t it supposed to be fixed?

[ingvaldlorentzen]

I think the main issue is that now the nginx.conf is generated from a Jinja template in the installation process.

If you're using a non-standard install process, you're probably gonna have to inject the nignx.conf file yourself. It's what I've done.

A simple COPY nginx.conf /etc/nginx/nginx.conf in a Dockerfile with the AWX image as the base does the trick.

This is the nginx.conf file I'm using, which is based on the Jinja template one:

#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

    server {

        listen 8052 default_server;

        # If you have a domain name, this is where to add it
        server_name _;
        keepalive_timeout 65;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;
            proxy_set_header X-Forwarded-Port 443;
        }
    }
}