bug: environment variable ${{ENV_VAR:=}} in apisix.yaml fails to resolve when using APISIX standalone mode in Docker

[kayx23]

Latest Update

Please see #10530 (comment)

Current Behavior

From the tests it seems that apisix.yaml supports the injection of environment variables.

Hence I previously opened #10509 for doc improvement. However, it's been noted from manual testing that env var in apisix.yaml did not get resolved when deploying APISIX in standalone mode in Docker.

Steps to Reproduce

docker run -d \
  --name apisix-standalone \
  -p9080:9080 -p9443:9443 -p9090:9092 \
  -e APISIX_STAND_ALONE=true \
  -e UPSTREAM_NODE_HTTPBIN="httpbin.org" \
  apache/apisix

Go into the container (docker exec -it apisix-standalone bash) and update apisix.yaml:

echo '
routes:
  -
    id: example-route-to-httpbin
    uri: /anything/test
    upstream:
      nodes:
        "${{UPSTREAM_NODE_HTTPBIN}}": 1
      type: roundrobin
#END
' > conf/apisix.yaml

Send a request to the route, if should not go through. Furthermore in logs you see failed to resolve variables:

Comments

Because these tests pass, I do suspect it might have something to do with the docker environment. However as seen the screenshot above, the env var was also correctly set in the container.

Environment

APISIX v3.7.0 Docker image

[Revolyssup]

@kayx23 While you exec and pass on the string with -c, the environment variable is resolved locally from the shell where you're executing the command and not from inside the container.

[kayx23]

For the above screenshot I actually went into the container and echo the configs into apisix.yaml. I only updated the command before creating this issue. Just edited it back. Sorry for the confusion.

[Revolyssup]

Can you try with ${} instead of ${{}}. I can assure that my env variable is getting picked up passed inside ${} @kayx23

[kayx23]

In yaml files? But all the docs and test cases suggest in config.yaml and apisix.yaml the env vars get picked up with the ${{...}} syntax. If the env var isn't set, there's even a syntax to support fallback values.

${..} is indeed what you would usually use in, say, a shell command, to resolve env vars. But I think it doesn't address the question here.

[JonathanDagan]

I'm also having the same issue using the 3.7.0 image and none of the following worked:

• $ENV://ENV_KEY as specified here
• ${{ENV_KEY}}
• ${{ENV_KEY:=}}
• ${ENV_KEY}

in all cases, I have made sure that the variable is loaded into the container either by using an outside env set (docker run -e or with docker-compose environment variables) to make sure that the environment variables where set.

From what I gathered from the logs there is the following error:

[lua] config_yaml.lua:112: failed: failed to resolve variables:failed to handle configuration: can't find environment variable REMOTE_API_HOST, context: ngx.timer

which indicates that even though the environment variable is set it needs some way to be loaded or be accessed through nginx explicitly

[kayx23]

cc: @monkeyDluffy6017 @shreemaan-abhishek

[hanqingwu]

@kayx23
i have trace this bug . so i find root cause is nginx ,

By default, nginx removes all environment variables inherited from its parent process except the TZ variable. This directive allows preserving some of the inherited variables, changing their values, or creating new environment variables.

The answer in the http://nginx.org/en/docs/ngx_core_module.html#env
so you need modify nginx.conf in apisix/cli/ngx_tpl.lua then add "env UPSTREAM_NODE_HTTPBIN;'"

then update apisix.yaml will work

some reference:
openresty/openresty#171

[kayx23]

Interesting... thanks for looking into it! @hanqingwu

I'll look into this a bit more later. I'm wondering why config.yaml doesn't have the same problem with env var (...or does it?). APISIX also has test coverage for this feature and the tests weren't failed...

[kayx23]

Just localized the issue a bit more and it doesn't seem to relate to @hanqingwu 's findings too much.

It has something to do with whether you export the env var or not:

• If you set the env var with export env_var=value, APISIX will pick up the env var normally.
• If you do not export and just set the env var with env_var=value, apisix init will fail with failed to resolve variables

I installed APISIX on the host machine and tested with config.yaml and apisix.yaml in standalone mode.

Successful Test with apisix.yaml

First, export UPSTREAM_NODE_HTTPBIN=httpbin.org

So I think this still has something to do with the way the environment variable is set for Docker.

[kayx23]

cc: @markokocic

[hanqingwu]

yes , if you first set apisix.yaml then start apisix , env var works.
but when you first start apisix , then set apisix.yaml , env var it fails . just like you initial steps as top .
these are my test result .
@kayx23

[markokocic]

Well, it still doesn't work with environment variables set in a standard docker way. The fact that docker is the standard and probably most common way of running Apisix, this should either be fixed or explicitly mentioned in documentation that docker -e or env files don't work.
CC: @kayx23 @AlinsRan

[kayx23]

Hi @markokocic , I updated the doc comfortable knowing that apsix.yaml does support env var, but perhaps shouldn't have closed this issue with the doc PR.

Based on @hanqingwu 's suggestions above, I suppose the next step could be to try to mount the apisix.yaml (with initial configurations that use env var) to the container at startup, though I haven't had the chance to give this a go.

We could reopen the issue if everyone thinks it's fitting.

[markokocic]

Hi @kayx23 , that was exactly what I tried in #10509 (comment)
apisix.yml is mounted and env variable set directly in docker compose file.

[kayx23]

@markokocic Ok I see. Could you share the complete docker compose file here if you still have it? We could update the issue description with it. Happy year end btw!

[shreemaan-abhishek]

@kayx23 I think we should document a reliable way to use ENV VARS in yaml for docker based deployments, what say?

[kayx23]

@shreemaan-abhishek can be documented if it works....

[markokocic]

hi @kayx23 here's my setup full example. I think we are a bit closer to finding out the underlying problem.

docker-compose-standalone.yml:

version: "3"

services:
  apisix:
    image: apache/apisix:${APISIX_IMAGE_TAG:-3.7.0-debian}
    restart: always
    volumes:
      - ./apisix_conf/apisix-standalone.yaml:/usr/local/apisix/conf/apisix.yaml:ro
    environment:
      - APISIX_STAND_ALONE=true
      - EP_MYHOST_1=b92.net
      - EP_MYHOST_2=blic.rs
    ports:
      - "9180:9180/tcp"
      - "9080:9080/tcp"
      - "9091:9091/tcp"
      - "9443:9443/tcp"
      - "9092:9092/tcp"
    networks:
      apisix:

  web1:
    image: nginx:1.19.0-alpine
    restart: always
    volumes:
      - ./upstream/web1.conf:/etc/nginx/nginx.conf
    ports:
      - "9081:80/tcp"
    environment:
      - NGINX_PORT=80
    networks:
      apisix:


networks:
  apisix:
    driver: bridge

apisix_conf/apisix-standalone.yaml:

routes:
  - uri: /b92/*
    upstream:
      nodes: 
        "${{EP_MYHOST_1:=}}:443": 1
      type: roundrobin
      scheme: https

#END

As you can see, I defined EP_MYHOST_1 and EP_MYHOST_2 environment variables in a standard docker way.

After that, I started docker with docker compose -f docker-compose-standalone.yml up -d

When I ran the following right after starting docker, I got success:

curl http://localhost:9080/b92/index.html -v

However, if I after that change the config file to use EP_MYHOST_2 instead of EP_MYHOST_1 like so:

routes:
  - uri: /b92/*
    upstream:
      nodes: 
        "${{EP_MYHOST_2:=}}:443": 1
      type: roundrobin
      scheme: https

#END

the config gets reloaded successfully, but the environment variable can't be resolved any more, although it is properly defined. If I edit it back to EP_MYHOST_1, it works again.

Looks like something gets wrong after reloading the config file. It seems like it can resolve env variables only if they are present in that file from the beginning, making the whole config reload feature confusing and not practical.

At the beginning, I was thinking environment variable resolution didn't work at all because I was starting with the empty config file and editing and reloading it incrementally, thinking it wouldn't matter if the variable was there from the start or not.

cc: @shreemaan-abhishek , @hanqingwu

[shreemaan-abhishek]

can be documented if it works....

#10530 (comment)

this works right?

EDIT: Or perhaps you tested with other installation method.

[markokocic]

hi @shreemaan-abhishek

If you look at my comments, both env variables are defined in the same standard docker way in docker compose.

However, looks like Apisix will resolve only variable that were present in apisix.yml when Apisix is started. The variables that are defined in docker compose, but added to apisix.yml after Apisix is started, will be somehow ignored after the config reload task.

[kayx23]

@shreemaan-abhishek the comment you took a screenshot of mentioned I directly installed on the host. I ran into a sed-related error starting APISIX in standalone mode with mounted volume, which blocked me from testing further: apache/apisix-docker#529

@markokocic 's latest comments are good summaries on the matter. It now seems to not be so Docker related and in line with what @hanqingwu described. Essentially, one would need to init again for nginx.conf to be updated.

The next step would be to evaluate whether this leads to only a doc improvement, or more.

[hanqingwu]

env directive should be used in nginx.conf. then os.getenv work well

it is a reliable way .

for reference:
https://github.com/openresty/lua-nginx-module#system-environment-variable-support

[shreemaan-abhishek]

However, looks like Apisix will resolve only variable that were present in apisix.yml when Apisix is started. The variables that are defined in docker compose, but added to apisix.yml after Apisix is started, will be somehow ignored after the config reload task.

Yes. You need to apisix reload after the config change. Because nginx doesn't inherit all the environment variables.

When apisix starts in standalone mode, it adds the env vars from apisix.yaml to the nginx.conf. And when you add new env vars in the apisix.yaml config while apisix is running, the they are added in nginx.conf only after apisix reload (or equivalent).

[markokocic]

Yes. You need to apisix reload after the config change. Because nginx doesn't inherit all the environment variables.

Shouldn't this be done automatically after detecting that config changed and before re-reading the config?

[shreemaan-abhishek]

reloading apisix everytime the config changes sounds like a very bad bad idea.

[shreemaan-abhishek]

I'll consider it resolved. #10530 (comment)

[shreemaan-abhishek]

I will close this once #10755 is merged.

[Vacant2333]

looks like we can close this issue now, thanks @shreemaan-abhishek