Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki 3.0.0: local ruler-storage now needlessly requires the rules directory to be writable #13222

Closed
whentze opened this issue Jun 14, 2024 · 2 comments · Fixed by #13248
Closed
Labels
type/bug Somehing is not working as expected

Comments

@whentze
Copy link

whentze commented Jun 14, 2024

Describe the bug
Loki v3.0.0, when configured to use the local ruler-storage backend and given a read-only rule directory, crashes at startup.

This regression seems to have been introduced in #12019, in which the EnsureDirectory utility was extended with an additional check that the directory be writable. That check is needed for the bloomstore but not for all the other users of that utility function, such as ruler-storage.

To Reproduce
Steps to reproduce the behavior:

  1. Configured local ruler-storage backend with a read-only directory
  2. Started loki v3.0.0
  3. It crashed (see below for log messages)

Expected behavior
Loki should start normally. The local ruler-storage backend is documented as read-only so it should work if that directory is read-only (as it did in 2.9.x).

Environment:

  • Infrastructure: Linux VM
  • Deployment tool: NixOS

Note: This issue also occurs in containerized setups in which the ruler storage is mounted in as a readonly volume.

Screenshots, Promtail config, or terminal output

insufficient permissions: /nix/store/1rnvm2hh2qmk37plaryxqpy8y2jzv2ri-loki-rules dr-xr-xr-x
error initialising module: ruler-storage
github.com/grafana/dskit/modules.(*Manager).initModule
        github.com/grafana/[email protected]/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
        github.com/grafana/[email protected]/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
        github.com/grafana/loki/v3/pkg/loki/loki.go:453
main.main
        github.com/grafana/loki/v3/cmd/loki/main.go:122
runtime.main
        runtime/proc.go:271
runtime.goexit
        runtime/asm_amd64.s:1695
@JStickler JStickler added the type/bug Somehing is not working as expected label Jun 17, 2024
owen-d added a commit to owen-d/loki that referenced this issue Jun 17, 2024
@owen-d
Copy link
Member

owen-d commented Jun 17, 2024

Ugh, sorry -- I pushed a PR to address this and make it less likely in the future.

@whentze
Copy link
Author

whentze commented Jun 18, 2024

No worries, and thank you for the very quick fix! 👍🏼

Could you cut a patch release for this, too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Somehing is not working as expected
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants