Permission denied on startup of Loki > 2.3.0 with docker volumes in docker-compose

[michaelosthege]

Describe the bug
When launching Loki in a docker-compose setup with a Docker volume to store the data, Loki runs into a permission denied error at startup.

To Reproduce
Steps to reproduce the behavior:

1. Take the docker-compose.yml from below with Loki version set to 2.3.0
2. Take the loki-config.yml from below
3. Run docker-compose up -d && watch docker ps -a and observe that Loki 2.3.0 starts as expected
4. Run docker-compose down && docker container rm db-loki && docker volume rm loki_data to stop the container and delete the volume
5. Change the docker-compose.yml to use Loki version 2.4.0
6. Run docker-compose up -d && watch docker ps -a and observe that Loki 2.4.0 dies right at the start

docker-compose.yml

version: "3.8"

volumes:
  loki_data:

services:
  db-loki:
    container_name: db-loki
    # working:
    #image: grafana/loki:2.3.0
    # broken:
    #image: grafana/loki:2.4.0
    volumes:
      - loki_data:/loki
      - ./config-loki.yml:/etc/loki/local-config.yaml
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

config-loki.yml

# This file is almost entirely just the default configuration file of Grafana Loki.
# The following customizations were made:
# + table_manager retention_period was changed

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  max_transfer_retries: 0     # Chunk transfers disabled

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

compactor:
  working_directory: /loki/boltdb-shipper-compactor
  shared_store: filesystem

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 720h

ruler:
  storage:
    type: local
    local:
      directory: /loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://localhost:9093
  ring:
    kvstore:
      store: inmemory
  enable_api: true

The logging output of the failed container shows the permission error:

level=info ts=2022-03-02T16:27:16.465251587Z caller=main.go:129 msg="Starting Loki" version="(version=, branch=, revision=)"
level=error ts=2022-03-02T16:27:16.557553034Z caller=log.go:106 msg="error running loki" err="mkdir /loki/chunks: permission denied\nerror creating index client\ngithub.jparrowsec.cn/cortexproject/cortex/pkg/chunk/storage.NewStore\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/chunk/storage/factory.go:198\ngithub.jparrowsec.cn/grafana/loki/pkg/loki.(*Loki).initStore\n\t/src/loki/pkg/loki/modules.go:322\ngithub.jparrowsec.cn/cortexproject/cortex/pkg/util/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:103\ngithub.jparrowsec.cn/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:75\ngithub.jparrowsec.cn/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:241\nmain.main\n\t/src/loki/cmd/loki/main.go:131\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nerror initialising module: store\ngithub.jparrowsec.cn/cortexproject/cortex/pkg/util/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:105\ngithub.jparrowsec.cn/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:75\ngithub.jparrowsec.cn/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:241\nmain.main\n\t/src/loki/cmd/loki/main.go:131\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"

Expected behavior
Starting Loki with storage in Docker volumes should work with any version.

Environment:

• OS: Ubuntu 20.04.3 LTS
• Docker version 20.10.12
• Docker-compose version 1.29.2

[dannykopping]

Hi @michaelosthege

I tried your above config, and could not replicate that error message.
What I did get instead was:

level=error ts=2022-03-03T06:43:39.330620824Z caller=log.go:106 msg="error running loki" err="mkdir wal: permission denied\nerror initialising module: ingester\ngithub.jparrowsec.cn/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.jparrowsec.cn/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:78\ngithub.jparrowsec.cn/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:285\nmain.main\n\t/src/loki/cmd/loki/main.go:96\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

The documentation is incorrect; the WAL is enabled by default:
https://github.com/grafana/loki/blob/main/pkg/ingester/wal.go#L44

If you add this under your ingester_config:

  wal:
    enabled: false

Then Loki will start up just fine.
Alternatively (and this is our recommendation), you should set the WAL dir:

  wal:
    dir: /loki/wal

[michaelosthege]

Hi @dannykopping, looks like I copied the wrong line after scrolling up in my terminal.

What is WAL ??
We followed the instructions from https://hub.docker.com/r/grafana/loki and they say nothing about WAL.

[dannykopping]

WAL is the "Write-Ahead Log" which the ingester uses to protect against data loss with unexpected crashes.
I'll make sure we update the docs on that page 👍

[dannykopping]

I updated the docs so we accurately reflect that the WAL is enabled by default as of v2.4.0:
https://grafana.com/docs/loki/latest/configuration/#ingester_config

[michaelosthege]

Hm, I'm sorry, but I fail to see how that solves the problem.
The feature is enabled by default, breaking any standard deployment with Docker volumes, and the recommendation is to turn it off?

I'm not familiar with the details of WAL and the resulting permission error, but to me it sounds like there's maybe a mkdir/chown thing missing in the creation of the Docker image?

[dannykopping]

The recommendation is to set the directory, not to turn it off:

Alternatively (and this is our recommendation), you should set the WAL dir:

We can't change anything in the docker image because otherwise the WAL will not be persisted. What we can do is update the docker instructions, which I'm busy trying to organise with my team.

[Depechie]

Sorry to hijack the thread, but @dannykopping seeing #1833 and #1949 I'm wondering where we are in how to use docker-compose with Loki 2.4.1.

Is this new mapping to /loki instead of /tmp/loki already live?
No matter what I try I also still get: mkdir /loki/rules: permission denied\nerror initialising module in the docker logs of Loki

Also most of the example configs still use the /tmp structure
https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml
https://github.com/grafana/loki/blob/main/docs/sources/configuration/examples/complete-local-config.yaml

[Thedude7054]

Is it inherently wrong to use the /tmp structure?

[hugoelopezp]

Was encountering this issue as well and realized the Loki container runs by default as user and group 10001. Solved the issue by changing the owner of the Loki bind to 10001:10001. This can also be resolved by passing the user and group of your preference as parameters when deploying the container.

Ref: https://grafana.com/docs/loki/latest/installation/docker/

[my-bam]

My solution:
Linux:

useradd -u 10001 loki
chown loki:loki /opt/loki/*

docker run:

-v /opt/loki:/tmp/loki

[shadow4040]

My solution: Linux:

useradd -u 10001 loki
chown loki:loki /opt/loki/*

docker run:

-v /opt/loki:/tmp/loki

thank you, only your solution is working, if we want use host directory instead volume directory

[myselfakashagarwal]

The giga chad solution that worked for me is to define user as root in compose

user: "root"

[techxiji]

sudo chmod 777 /loki