Cannot log in / password not working before and after batch-reset

[hamsdac]

Problem description

Issue
After bringing up the stack with docker-compose up -d, the default password changeme does not let me login with user elastic.

Expected outcome
Logging in with elastic / changeme should work

Extra information

After changing the password with the docker-compose ... bin/elasticsearch-reset-password --batch ... command, then writing the new values in .env file and restarting the containers, kibana never reaches the "ready" state.
Also, after doing this, I see the following logs:

elasticsearch_1  | {"@timestamp":"2022-04-08T07:28:32.233Z", "log.level": "WARN", "message":"Authentication to realm default_native failed - Password authentication failed for logstash_internal", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a9e8eae67e4c][system_critical_read][T#3]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"FJTpuyL1ReygWuVzDXkrdg","elasticsearch.node.id":"LYKLOcPrTZyQvq4Qtik94w","elasticsearch.node.name":"a9e8eae67e4c","elasticsearch.cluster.name":"docker-cluster"}

elasticsearch_1  | {"@timestamp":"2022-04-08T07:28:24.637Z", "log.level": "INFO", "message":"Authentication of [kibana_system] was terminated by realm [reserved] - failed to authenticate user [kibana_system]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a9e8eae67e4c][system_critical_read][T#2]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","trace.id":"57bea88ec5a71670b54fde9db23d4121","elasticsearch.cluster.uuid":"FJTpuyL1ReygWuVzDXkrdg","elasticsearch.node.id":"LYKLOcPrTZyQvq4Qtik94w","elasticsearch.node.name":"a9e8eae67e4c","elasticsearch.cluster.name":"docker-cluster"}

elasticsearch_1  | {"@timestamp":"2022-04-08T07:28:31.782Z", "log.level": "INFO", "message":"Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a9e8eae67e4c][system_critical_read][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.RealmsAuthenticator","elasticsearch.cluster.uuid":"FJTpuyL1ReygWuVzDXkrdg","elasticsearch.node.id":"LYKLOcPrTZyQvq4Qtik94w","elasticsearch.node.name":"a9e8eae67e4c","elasticsearch.cluster.name":"docker-cluster"}

Stack configuration

Changes to docker-compose.yml:

# changes to service elasticsearch
-      ES_JAVA_OPTS: -Xmx256m -Xms256m
+      ES_JAVA_OPTS: -Xmx1024m -Xms1024m

# changes to service logstash
-      - "5000:5000/tcp"
-      - "5000:5000/udp"
+      - "5001:5000/tcp"
+      - "5001:5000/udp"

I needed to change the logstash ports because port 5000 is in use and I cannot change the other application.

Other changes:
license set to basic before first start

Docker setup

$ docker version

Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:02 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:53 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker-compose version

docker-compose version 1.25.0, build unknown
docker-py version: 4.1.0
CPython version: 3.8.10
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Container logs

Pastebin (had to truncate pastebin, was too long)

[antoineco]

@hamsdac thanks for the detailed issue report 🙌

Officially we only support Compose v1.26 and above: https://github.com/deviantony/docker-elk#host-setup.
You are using v1.25.

With that being said, v1.25 is known to work too, with one quirk: quotes aren't removed around the values inside .env files. That means Compose is literally setting ELASTIC_PASSWORD to "changeme" (quotes included). (ref. #677 (comment))

To fix that, I recommend either removing quotes around values inside the .env file (but be careful if you passwords contain special characters) or updating Compose.

[hamsdac]

Thank you very much, that was the problem.
I replaced the changeme in .env with long passwords without special characters (only a-z / A-Z / 0-9), removed the ' around the passwords and now it's working fine :)

EDIT
FIY: I did not update docker-ce or docker-compose, as no updates were available for Ubuntu 20.04 with official docker repos.

[antoineco]

As long as you're aware of this counterintuitive behaviour with quotes on Compose versions between v1.22.0 and v1.25.5 you should be good to go 👍