Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bloom gateway panic: runtime error: index out of range [0] with length 0 while fetching chunk refs #14623

Closed
flaneur23 opened this issue Oct 25, 2024 · 1 comment · Fixed by #14634
Closed

Bloom gateway panic: runtime error: index out of range [0] with length 0 while fetching chunk refs #14623

flaneur23 opened this issue Oct 25, 2024 · 1 comment · Fixed by #14634
Labels
backport release-3.2.x feature/blooms type/bug Somehing is not working as expected

Comments

@flaneur23
Copy link

flaneur23 commented Oct 25, 2024 •
edited
Loading

Describe the bug
A clear and concise description of what the bug is.
I checked why I'm facing high latency while running queries like {service_name="some_service"} |= "trace_id=67187c7567b5e32e1f03fce3a0fa57bf". It turned out that most of the time is spent on downloading chunk refs.

| total_time  14s | queued   0% | execution   0% | index 100% | store   0% |

What I found in bloom gateway logs is:

2024-10-25 18:31:37.534	panic: runtime error: index out of range [0] with length 0
2024-10-25 18:31:37.534	goroutine 5182 [running]:
2024-10-25 18:31:37.534	github.com/grafana/loki/v3/pkg/util/server.onPanic({0x3547aa0, 0xc011f6b0b0})
2024-10-25 18:31:37.534		/src/loki/pkg/util/server/recovery.go:57 +0x4d
2024-10-25 18:31:37.534	github.com/grafana/loki/v3/pkg/util/server.init.WithRecoveryHandler.func4.1({0xc00270b408?, 0x0?}, {0x3547aa0?, 0xc011f6b0b0?})
2024-10-25 18:31:37.534		/src/loki/vendor/github.com/grpc-ecosystem/go-grpc-middleware/recovery/options.go:33 +0x27
2024-10-25 18:31:37.534	github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x42e8230?, 0xc0061f24e0?}, {0x3547aa0?, 0xc011f6b0b0?}, 0xc0079ee600?)
2024-10-25 18:31:37.534		/src/loki/vendor/github.com/grpc-ecosystem/go-grpc-middleware/recovery/interceptors.go:61 +0x30
2024-10-25 18:31:37.534	github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
2024-10-25 18:31:37.534		/src/loki/vendor/github.com/grpc-ecosystem/go-grpc-middleware/recovery/interceptors.go:29 +0x75
2024-10-25 18:31:37.534	panic({0x3547aa0?, 0xc011f6b0b0?})
2024-10-25 18:31:37.534		/usr/local/go/src/runtime/panic.go:770 +0x132
2024-10-25 18:31:37.534	github.com/grafana/loki/v3/pkg/bloomgateway.(*Gateway).FilterChunkRefs(0xc001299808, {0x42e8230, 0xc0061f24e0}, 0xc0061d39a0)
2024-10-25 18:31:37.534		/src/loki/pkg/bloomgateway/bloomgateway.go:241 +0x16e7
2024-10-25 18:31:37.534	github.com/grafana/loki/v3/pkg/logproto._BloomGateway_FilterChunkRefs_Handler.func1({0x42e8230?, 0xc0061f24e0?}, {0x370a1a0?, 0xc0061d39a0?})
2024-10-25 18:31:37.534		/src/loki/pkg/logproto/bloomgateway.pb.go:546 +0xcb
2024-10-25 18:31:37.534	github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x42e8230?, 0xc0061f24e0?}, {0x370a1a0?, 0xc0061d39a0?}, 0x42e8230?, 0xc0061f24b0?)
2024-10-25 18:31:37.534		/src/loki/vendor/github.com/grpc-ecosystem/go-grpc-middleware/recovery/interceptors.go:33 +0xb0
2024-10-25 18:31:37.534	google.golang.org/grpc.getChainUnaryHandler.func1({0x42e8230, 0xc0061f24e0}, {0x370a1a0, 0xc0061d39a0})
2024-10-25 18:31:37.534		/src/loki/vendor/google.golang.org/grpc/server.go:1196 +0xb2
2024-10-25 18:31:37.534	github.com/grafana/dskit/middleware.ServerUserHeaderInterceptor({0x42e8230?, 0xc0061f24b0?}, {0x370a1a0, 0xc0061d39a0}, 0x40?, 0xc0061ee640)
2024-10-25 18:31:37.534		/src/loki/vendor/github.com/grafana/dskit/middleware/grpc_auth.go:43 +0x5b
2024-10-25 18:31:37.534	github.com/grafana/loki/v3/pkg/util/fakeauth.SetupAuthMiddleware.func1({0x42e8230, 0xc0061f24b0}, {0x370a1a0, 0xc0061d39a0}, 0xc00636d0c0, 0xc0061ee640)
2024-10-25 18:31:37.534		/src/loki/pkg/util/fakeauth/fake_auth.go:27 +0x9d
2024-10-25 18:31:37.534	google.golang.org/grpc.getChainUnaryHandler.func1({0x42e8230, 0xc0061f24b0}, {0x370a1a0, 0xc0061d39a0})
2024-10-25 18:31:37.534		/src/loki/vendor/google.golang.org/grpc/server.go:1196 +0xb2
2024-10-25 18:31:37.535	github.com/grafana/dskit/middleware.UnaryServerInstrumentInterceptor.func1({0x42e8230, 0xc0061f24b0}, {0x370a1a0, 0xc0061d39a0}, 0xc00636d0c0, 0xc0061ee600)
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/grafana/dskit/middleware/grpc_instrumentation.go:46 +0xbd
2024-10-25 18:31:37.535	google.golang.org/grpc.getChainUnaryHandler.func1({0x42e8230, 0xc0061f24b0}, {0x370a1a0, 0xc0061d39a0})
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1196 +0xb2
2024-10-25 18:31:37.535	github.com/grafana/dskit/server.newServer.HTTPGRPCTracingInterceptor.func3({0x42e8230?, 0xc0061f24b0?}, {0x370a1a0?, 0xc0061d39a0?}, 0xc00636d0c0?, 0xc012d414d0?)
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/grafana/dskit/middleware/http_tracing.go:74 +0x97c
2024-10-25 18:31:37.535	google.golang.org/grpc.getChainUnaryHandler.func1({0x42e8230, 0xc0061f24b0}, {0x370a1a0, 0xc0061d39a0})
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1196 +0xb2
2024-10-25 18:31:37.535	github.com/opentracing-contrib/go-grpc.OpenTracingServerInterceptor.func1({0x42e8230, 0xc0061f2390}, {0x370a1a0, 0xc0061d39a0}, 0xc00636d0c0, 0xc0061ee580)
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/opentracing-contrib/go-grpc/server.go:57 +0x3e7
2024-10-25 18:31:37.535	google.golang.org/grpc.getChainUnaryHandler.func1({0x42e8230, 0xc0061f2390}, {0x370a1a0, 0xc0061d39a0})
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1196 +0xb2
2024-10-25 18:31:37.535	github.com/grafana/dskit/middleware.GRPCServerLog.UnaryServerInterceptor({{0x42ab2c0?, 0xc0000ee7d0?}, 0x0?, 0xd?}, {0x42e8230, 0xc0061f2390}, {0x370a1a0, 0xc0061d39a0}, 0xc00636d0c0, 0xc0061ee540)
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/grafana/dskit/middleware/grpc_logging.go:54 +0xaf
2024-10-25 18:31:37.535	google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x42e8230, 0xc0061f2390}, {0x370a1a0, 0xc0061d39a0}, 0xc00636d0c0, 0x80?)
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1187 +0x85
2024-10-25 18:31:37.535	github.com/grafana/loki/v3/pkg/logproto._BloomGateway_FilterChunkRefs_Handler({0x35f4060, 0xc001299808}, {0x42e8230, 0xc0061f2390}, 0xc013324400, 0xc001a1c420)
2024-10-25 18:31:37.535		/src/loki/pkg/logproto/bloomgateway.pb.go:548 +0x143
2024-10-25 18:31:37.535	google.golang.org/grpc.(*Server).processUnaryRPC(0xc0013be000, {0x42e8230, 0xc0061f22a0}, {0x4309060, 0xc002732600}, 0xc0061dc120, 0xc00286ce40, 0x6272540, 0x0)
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1379 +0xdf8
2024-10-25 18:31:37.535	google.golang.org/grpc.(*Server).handleStream(0xc0013be000, {0x4309060, 0xc002732600}, 0xc0061dc120)
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1790 +0xe8b
2024-10-25 18:31:37.535	google.golang.org/grpc.(*Server).serveStreams.func2.1()
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1029 +0x8b
2024-10-25 18:31:37.535	created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 5172
2024-10-25 18:31:37.535		/src/loki/vendor/google.golang.org/grpc/server.go:1040 +0x125
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 1 [select, 21 minutes]:
2024-10-25 18:31:37.535	github.com/grafana/dskit/services.(*Manager).AwaitStopped(0xc002a96d20, {0x42e8020, 0x6355480})
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/grafana/dskit/services/manager.go:153 +0x67
2024-10-25 18:31:37.535	github.com/grafana/loki/v3/pkg/loki.(*Loki).Run(0xc00248a008, {0x0?, {0x4?, 0x2?, 0x62ee420?}})
2024-10-25 18:31:37.535		/src/loki/pkg/loki/loki.go:592 +0xea5
2024-10-25 18:31:37.535	main.main()
2024-10-25 18:31:37.535		/src/loki/cmd/loki/main.go:129 +0x1333
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 8 [select, 21 minutes]:
2024-10-25 18:31:37.535	github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:375 +0xa5
2024-10-25 18:31:37.535	created by github.com/baidubce/bce-sdk-go/util/log.NewLogger in goroutine 1
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/baidubce/bce-sdk-go/util/log/logger.go:368 +0x116
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 9 [select]:
2024-10-25 18:31:37.535	go.opencensus.io/stats/view.(*worker).start(0xc0006fe500)
2024-10-25 18:31:37.535		/src/loki/vendor/go.opencensus.io/stats/view/worker.go:292 +0x9f
2024-10-25 18:31:37.535	created by go.opencensus.io/stats/view.init.0 in goroutine 1
2024-10-25 18:31:37.535		/src/loki/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 43 [syscall, 21 minutes]:
2024-10-25 18:31:37.535	os/signal.signal_recv()
2024-10-25 18:31:37.535		/usr/local/go/src/runtime/sigqueue.go:152 +0x29
2024-10-25 18:31:37.535	os/signal.loop()
2024-10-25 18:31:37.535		/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
2024-10-25 18:31:37.535	created by os/signal.Notify.func1.1 in goroutine 324
2024-10-25 18:31:37.535		/usr/local/go/src/os/signal/signal.go:151 +0x1f
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 200 [chan receive]:
2024-10-25 18:31:37.535	github.com/grafana/loki/v3/pkg/util/log.newPrometheusLogger.WithFlushPeriod.func2.1()
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/grafana/dskit/log/buffered.go:76 +0x97
2024-10-25 18:31:37.535	created by github.com/grafana/loki/v3/pkg/util/log.newPrometheusLogger.WithFlushPeriod.func2 in goroutine 1
2024-10-25 18:31:37.535		/src/loki/vendor/github.com/grafana/dskit/log/buffered.go:72 +0x65
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 241 [select, 2 minutes]:
2024-10-25 18:31:37.535	github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.(*BlocksCache).runTTLEvictJob(0xc0006fec00, 0xdf8475800, 0x4e94914f0000)
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/blockscache.go:430 +0x1c5
2024-10-25 18:31:37.535	created by github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.NewFsBlocksCache in goroutine 1
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/blockscache.go:148 +0x20e
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 242 [select, 21 minutes]:
2024-10-25 18:31:37.535	github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.(*BlocksCache).runLRUevictJob(0xc0006fec00)
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/blockscache.go:386 +0x125
2024-10-25 18:31:37.535	created by github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.NewFsBlocksCache in goroutine 1
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/blockscache.go:149 +0x24a
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 243 [select]:
2024-10-25 18:31:37.535	github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.(*BlocksCache).runMetricsCollectJob(0xc0006fec00, 0x12a05f200)
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/blockscache.go:374 +0x193
2024-10-25 18:31:37.535	created by github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.NewFsBlocksCache in goroutine 1
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/blockscache.go:150 +0x286
2024-10-25 18:31:37.535	
2024-10-25 18:31:37.535	goroutine 270 [select]:
2024-10-25 18:31:37.535	github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.newCachedListOpObjectClient.func1(0xc00285a550)
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/client.go:477 +0xdd
2024-10-25 18:31:37.535	created by github.com/grafana/loki/v3/pkg/storage/stores/shipper/bloomshipper.newCachedListOpObjectClient in goroutine 1
2024-10-25 18:31:37.535		/src/loki/pkg/storage/stores/shipper/bloomshipper/client.go:472 +0x105

I'm not sure if the stack trace is enough to debug it further. If not let me know if You need something from config, more context, etc.

To Reproduce
Steps to reproduce the behavior:
Not relevant

Expected behavior
A clear and concise description of what you expected to happen.

Run bloom filtered query without issues.

Environment:

  • Loki ver: 3.2.0
  • Infrastructure: Kubernetes
  • Deployment tool: helm (distributed mode)

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.
@chaudum chaudum mentioned this issue Oct 29, 2024
Merged
6 tasks
@chaudum
Copy link
Contributor

chaudum commented Oct 29, 2024

@flaneur23 Thanks for reporting!

@chaudum chaudum added backport release-3.2.x type/bug Somehing is not working as expected labels Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backport release-3.2.x feature/blooms type/bug Somehing is not working as expected
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(blooms): Check length of tasks before accessing first element in slice grafana/loki
2 participants
@chaudum @flaneur23