Support multiple .env files

[aurelijusrozenas]

It's pretty common to have multiple .env files to override settings and is used in many projects. Some of the examples would be:

• project may be configured for dev (.env.dev) and prod (.env.prod) stages and have general configuration applied to both stages (.env).
• project may have standard configuration in .env and override some settings with .env.local for each developer.

Multiple file support is implemented for docker run and as env_file docker-compose property and it works well for environment variables in docker container. However different environment variables are very useful in docker-compose.yaml too. E.g. I have a project where url is setup via labels and I do have different urls for prod and dev stages.

Right now I think the only way to have separated dev and prod stages is having two files with all options in both them but that is not DRY and error prone when adding/changing values.

[shaunco]

Seems related to #7289

[1oglop1]

@shaunco #7289 was closed for being stale, is there any alternative?
It's becoming pretty painful in our dev flow

[xdevs23]

I'd like to see this as well.

[ndeloof]

You can select dotEnv file to be used by Compose using docker compose --env-file xxx youcommand
We could extend profile support to automatically select an alternate dotEnv file, but I wonder this would trigger too much unexpected side effects to use who already have sibling dot env file in their project to be passed to containers (not compose).

[xdevs23]

Perhaps it would be sufficient to just specify the env files in docker-compose.yaml which is already supported:

# ...
env_file:
  # whichever you prefer
  - ${ENV}.env
  - .env.${ENV}

And just specify ENV when running the command.

[ndeloof]

env_file in compose.yaml is the one used to define container environment, not the one used to parse yaml and interpolate $xx variables.

[frob]

You can select dotEnv file to be used by Compose using docker compose --env-file xxx youcommand
We could extend profile support to automatically select an alternate dotEnv file, but I wonder this would trigger too much unexpected side effects to use who already have sibling dot env file in their project to be passed to containers (not compose).

How about just allowing for multiple --env-file arguments to be parsed? Then it would be user choice docker compose --env-file .env up or docker compose --env-file .env --env-file .env.prod up etc.

[designermonkey]

This is a real pain that I've come across also, however, I got a little creative in the meantime to try and work around it. I hope this may be helpful to someone else.

I use makefiles to make the repetitive tying of commands easier, which also allows some extra abilities for things like env vars:

.EXPORT_ALL_VARIABLES:
RED=$(shell tput setaf 1)
RESET=$(shell tput sgr0)


# Specify some legal environments and have a variable with a default
envs = dev,tst,prd
ENV ?= dev

# Throw if not a legal environment
ifeq (, $(findstring $(ENV),$(envs)))
$(error "$(RED)Provided ENV '$(ENV)' is not a valid environment. One of $(envs) must be used.$(RESET)")
endif


# Define some env file paths
defaults = .env.defaults
env = .env
dep = .env.$(ENV)


# Include (without erroring) those file paths
-include $(defaults)
export $(shell sed 's/=.*//' $(defaults) > /dev/null 2>&1)

-include $(env)
export $(shell sed 's/=.*//' $(env) > /dev/null 2>&1)


# Write out the commands, notice the dep env file is loaded here to ensure
# it is not confused by the normal .env file which we already loaded
up:
	docker compose --env-file=$(dep) up

Please bear in mind that the above is not tested as it's not exactly how I do things, but I thought it might help as an example of what can be done.

[aurelijusrozenas]

So, couple of years later, the problem is still big pain when setuping dev projects. .env is usually committed to repository with global settings and there is .env.local file to override something if there is a need for that. With docker-compose it is not possible to pass both files so you need to write some kind of hacky script to maybe make it work. I cannot think of any other way.

[aurelijusrozenas]

Sooo, if you happen to need a hacky script there you go ;)

#!/usr/bin/env bash

##
# docker-compose bin wrapper that loads `.env` if found (default functionality) and additionally overrides values from `.env.local` if exists.
# Does not (and must not) produce additional output so that the PhpStorm could use it as docker-compose replacement.
# Also script name must be exactly `docker-compose` for the PhpStorm.
##

# exit script on first failed command or unset variable and if pipeline fails
set -euo pipefail
# echo on
#set -x

# if `--env-file` was not passed, load .env and .env.local
if [[ "$*" != *"--env-file"* ]]; then
    set -o allexport
    if [[ -f .env ]]; then
        source .env
    fi
    if [[ -f .env.local ]]; then
        source .env.local
    fi
    set +o allexport
fi

# `"$@"` is very important, it ensures that arguments are split and quoted correctly when passing quoted argument with space in the middle
# https://stackoverflow.com/a/10836225/846432
/usr/local/bin/docker-compose "$@"

I was able to successfully use it from terminal, just add it to $PATH. If you name the script docker-compose autocomplete will work without any changes.
After some poking I managed to use it in PhpStorm also. There you need to set Build, Execution, Deployment | Docker | Tools -> Docker compose executable to you script path. The script must be named docker-compose.

So as far as I am concerned I completely solved my problem. When docker-compose will add support for multiple env files I will still need to keep a wrapper so that my logic for .env and .env.local would still work.

How to adapt this script for you case?

1. There is hardcoded path to original script /usr/local/bin/docker-compose but it is the same for linux and macos so it should be fine for most cases.
2. File loading is now hardcoded so if you need to load different files just change paths. You want to pass them via CLI with multiple --env-file - well then you must do some bash voodoo magic and share your solution with others ;)

[kasir-barati]

--env-file .env --env-file

@frob This command for me does not work: docker-compose -f next.docker-compose.yml --env-file .env --env-file .redis.env --env-file .postgresql.env up --build

[themagic314]

@frob This command for me does not work: docker-compose -f next.docker-compose.yml --env-file .env --env-file .redis.env --env-file .postgresql.env up --build

It's not implemented. "How about just allowing..."

[nemanjam]

You can select dotEnv file to be used by Compose using docker compose --env-file xxx youcommand
We could extend profile support to automatically select an alternate dotEnv file, but I wonder this would trigger too much unexpected side effects to use who already have sibling dot env file in their project to be passed to containers (not compose).

How about just allowing for multiple --env-file arguments to be parsed? Then it would be user choice docker compose --env-file .env up or docker compose --env-file .env --env-file .env.prod up etc.

I can't just pass multiple .env.* files? All vars must be in a single file?

[kasir-barati]

@nemanjam I asked your same question and the answer is nope, It is not implemented unfortunately.

[a1exus]

and yet per https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option it says:

You can pass multiple environment variables from an external file through to a service’s containers with the ‘env_file’ option, just like with docker run --env-file=FILE ...:

yet i see it doesn't work, even though multiple files are specified in "env_file"

[kasir-barati]

@a1exus Check my docker repo and see what are you doing wrongly. Note I have multiple branch and this is the docker-compose branch. I hope it can help you. Do not hesitate to contribute and create pull request if you wanted.

[kashev]

@a1exus The documentation you're linking relates to passing environment variables to the containers themselves, as mentioned in this thread here.

[samuelvi]

"Merging" several .env files "on-the-fly" is possible by passing the content via stdin to the arg --env-file:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" "./docker/.env.local") up -d

And if you want to check whether the .env.* files exists or not:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" && ([ -f "./docker/.env.local" ] && cat "./docker/.env.local" || echo '')) up -d

Hope it helps :)

[kingyue737]

@samuelvi Does not work for me, warning shows that no env variables are loaded:

WARN[0000] The "PGADMIN_DEFAULT_EMAIL" variable is not set. Defaulting to a blank string.
WARN[0000] The "PGADMIN_DEFAULT_PASSWORD" variable is not set. Defaulting to a blank string.
WARN[0000] The "PGADMIN_HOST_PORT" variable is not set. Defaulting to a blank string.
...

I tried the following minimal example:

docker-compose --env-file <(cat "./.env") up -d

docker-compose version v2.12.2

Any help would be appreciated!

[misuzu]

docker-compose --env-file <(cat *.env) up would be perfect, but it doesn't work.

[ghz-max]

I was able to make it work with:

export $(cat "/opt/env" "./app.env"| xargs) && docker compose up

[lexalium]

"Merging" several .env files "on-the-fly" is possible by passing the content via stdin to the arg --env-file:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" "./docker/.env.local") up -d

And if you want to check whether the .env.* files exists or not:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" && ([ -f "./docker/.env.local" ] && cat "./docker/.env.local" || echo '')) up -d

Hope it helps :)

It works for me only with old docker compose version (v2.2.2).
It doesn't work with new version. Tried on v2.14.2.

[add-n2x]

It would be really helpful to have a global definition of env_file in Docker Compose where you specifiy all dot.env files to be loaded:

env_file:
  - .env
  - .env.version
  - .env.local

[ndeloof]

@davidtrattnig if you don't want to repeat yourself you can use yaml anchors:

x-env:
  &common_env_files
  - .env
  - .env.version
  - .env.local

services:
  foo:
    env_file: *common_env_files

[ndeloof]

after #10284 --env-file flag can be used in multiple occurrences to select more than one env file.

[kaatula]

after #10284 --env-file flag can be used in multiple occurrences to select more than one env file.

Shouldn't this be mentioned in the Release Notes? And docs?

[ndeloof]

indeed. Release note is generated from pull requests description, but here multiple changes/features were introduced
fixed

[VictorSCamargo]

I could finally make multiple --env-file work. The problem seems to be related with docker-compose being installed. Docker already has "docker compose" that seems to be the same as "docker-compose", but maybe there was a conflict. Fully uninstalling docker-compose and docker, and installing docker again made passing multiple --env-file work. Like the example "docker compose --env-file a.env --env-file b.env"

[izakdvlpr]

"Merging" several .env files "on-the-fly" is possible by passing the content via stdin to the arg --env-file:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" "./docker/.env.local") up -d

And if you want to check whether the .env.* files exists or not:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" && ([ -f "./docker/.env.local" ] && cat "./docker/.env.local" || echo '')) up -d

Hope it helps :)

Thanks!

[karand1985]

Hi all, the support for multiple environment files is part of which docker compose version?
I have docker version as 20.10.17 and docker compose version as v2.6.0

Regards.

[kasir-barati]

@karand1985 I think it is not supported the way that I think you expect it to be implemented. Just use this trick instead: https://github.com/kasir-barati/docker/tree/docker-compose?tab=readme-ov-file#tips

[andreworg]

I think this issue should not have been closed before the feature was also supported in yml, which as of today is still not the case AFAICS.

[ndeloof]

@andreworg mutiple env_file IS supported in compose.yaml for long time

[andreworg]

@andreworg mutiple env_file IS supported in compose.yaml for long time

It is actually supported as a service top-leve element: https://docs.docker.com/compose/compose-file/05-services/#env_file
but not as a top-level element, which is how the cli switch works; at least as far as I can see.

I also tried using include with the "long syntax": https://docs.docker.com/compose/compose-file/14-include/#long-syntax
but I could not make it work. I believe the env_file you reference in the include is only applied to the relevant sub-project, which makes sense.

[ndeloof]

but not as a top-level element

there's no env_file top-level element. env-file is a command line flag, used to parse the compose file. Having this declared inside the compose.yaml would bring a chicken-egg issue

I believe the env_file you reference in the include is only applied to the relevant sub-project,

indeed. include is basically "please get compose model from the equivalent docker compose config for this sibling project"

[ianhinder]

I have been able to use include to supply multiple env files for use in interpolation in the compose file.

$ docker compose version
Docker Compose version v2.29.2-desktop.2

$ cat docker-compose.real.yml 
services:
  helloworld:
    image: alpine:latest
    command: echo "${GREETING} ${WHERE}!"

$ cat docker-compose.yml 
include:
  - path: docker-compose.real.yml
    env_file:
       - .env.common
       - .env.local

$ cat .env.common 
GREETING=Hello

$ cat .env.local 
WHERE=World

$ docker compose run --rm helloworld
Hello World!

Note one potential gotcha: any file called ".env" will override anything that is supplied in the provided env_file files. So if your first env_file is called .env, which would not be unusual, it will actually be applied again at the end, making it look like the files are applied in the wrong order.

Suggestions for a better name than docker-compose.real.yml welcome!

[petrosmm]

"Merging" several .env files "on-the-fly" is possible by passing the content via stdin to the arg --env-file:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" "./docker/.env.local") up -d

And if you want to check whether the .env.* files exists or not:

docker-compose -f "./docker/docker-compose.yaml" --env-file <(cat "./docker/.env" && ([ -f "./docker/.env.local" ] && cat "./docker/.env.local" || echo '')) up -d

Hope it helps :)

This looks awesome and works manually on command line but inside of a standard .sh script it reports an error. Does anyone know why?

[ianhinder]

Is it a "/bin/sh" script or a "/bin/bash" script? the process redirection feature <(command) is a bash-specific feature; it is not supported in /bin/sh.