Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiline parser does not work for Containerd #4508

Closed
xiaoping8385 opened this issue Dec 22, 2021 · 9 comments
Closed

multiline parser does not work for Containerd #4508

xiaoping8385 opened this issue Dec 22, 2021 · 9 comments
Labels
fixed Stale

Comments

@xiaoping8385
Copy link

xiaoping8385 commented Dec 22, 2021 •
edited
Loading

Bug Report

Describe the bug
Hi there, I configure my fluent-bit as :

[INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/test.log
        DB                /var/log/flb_kube.db
        multiline.parser  docker, cri
[FILTER]
        Name                   multiline
        Match                  *
        multiline.key_content  log
        multiline.parser       go, java

and
the initial container.log is like ：

Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)
another line...
panic: my panic

goroutine 4 [running]:
panic(0x45cb40, 0x47ad70)
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
main.main.func1(0xc420024120)
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
created by main.main
  foo.go:5 +0x58

goroutine 1 [chan receive]:
runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)
  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c
runtime.goparkunlock(0xc420024178, 0x46fcd7, 0xc, 0x1000f010040c217, 0x3)
  /usr/local/go/src/runtime/proc.go:286 +0x5e fp=0xc420053e70 sp=0xc420053e30 pc=0x42512e
runtime.chanrecv(0xc420024120, 0x0, 0xc420053f01, 0x4512d8)
  /usr/local/go/src/runtime/chan.go:506 +0x304 fp=0xc420053f20 sp=0xc420053e70 pc=0x4046b4
runtime.chanrecv1(0xc420024120, 0x0)

which I copied from the official doc sample.

As I was running it in containerd container, so the output log under /var/log/containers/ is like

2021-12-22T02:36:56.941050368Z stdout F single line...
2021-12-22T02:36:56.941096311Z stdout F Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
2021-12-22T02:36:56.941101802Z stdout F     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
2021-12-22T02:36:56.9411043Z stdout F     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
2021-12-22T02:36:56.94110682Z stdout F     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
2021-12-22T02:36:56.941109302Z stdout F     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
2021-12-22T02:36:56.94111173Z stdout F     at com.myproject.module.MyProject.main(MyProject.java:6)
2021-12-22T02:36:56.941114604Z stdout F another line...
2021-12-22T02:36:56.941116989Z stdout F panic: my panic
2021-12-22T02:36:56.94111914Z stdout F 
2021-12-22T02:36:56.941122703Z stdout F goroutine 4 [running]:
2021-12-22T02:36:56.941125769Z stdout F panic(0x45cb40, 0x47ad70)
2021-12-22T02:36:56.941128769Z stdout F   /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
2021-12-22T02:36:56.941131055Z stdout F main.main.func1(0xc420024120)
2021-12-22T02:36:56.941133533Z stdout F   foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
2021-12-22T02:36:56.941135853Z stdout F runtime.goexit()
2021-12-22T02:36:56.941138281Z stdout F   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
2021-12-22T02:36:56.941140724Z stdout F created by main.main
2021-12-22T02:36:56.941142941Z stdout F   foo.go:5 +0x58
2021-12-22T02:36:56.941145054Z stdout F 
2021-12-22T02:36:56.9411473

I expected to see those mulitiline logs be concatenated but I only get the first mulitiline message been successfully handled and not the next.
the output was like:

Fluent Bit v1.8.10
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[0] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116297444, {"time"=>"2021-12-22T02:05:41.116297444Z", "stream"=>"stdout", "_p"=>"F", "log"=>"single line..."}]
[1] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116334376, {"time"=>"2021-12-22T02:05:41.116334376Z", "stream"=>"stdout", "_p"=>"F", "log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)"}]
[2] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116360318, {"time"=>"2021-12-22T02:05:41.116360318Z", "stream"=>"stdout", "_p"=>"F", "log"=>"another line..."}]
[3] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116363462, {"time"=>"2021-12-22T02:05:41.116363462Z", "stream"=>"stdout", "_p"=>"F", "log"=>"panic: my panic"}]
[4] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138743.404566405, {"log"=>"2021-12-22T02:05:41.116366563Z stdout F "}]
[5] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116370499, {"time"=>"2021-12-22T02:05:41.116370499Z", "stream"=>"stdout", "_p"=>"F", "log"=>"goroutine 4 [running]:"}]
[6] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116374103, {"time"=>"2021-12-22T02:05:41.116374103Z", "stream"=>"stdout", "_p"=>"F", "log"=>"panic(0x45cb40, 0x47ad70)"}]
[7] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116378042, {"time"=>"2021-12-22T02:05:41.116378042Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c"}]
[8] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116381702, {"time"=>"2021-12-22T02:05:41.116381702Z", "stream"=>"stdout", "_p"=>"F", "log"=>"main.main.func1(0xc420024120)"}]
[9] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116385381, {"time"=>"2021-12-22T02:05:41.116385381Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339"}]
[10] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116388793, {"time"=>"2021-12-22T02:05:41.116388793Z", "stream"=>"stdout", "_p"=>"F", "log"=>"runtime.goexit()"}]
[11] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116392343, {"time"=>"2021-12-22T02:05:41.116392343Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1"}]
[12] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116395781, {"time"=>"2021-12-22T02:05:41.116395781Z", "stream"=>"stdout", "_p"=>"F", "log"=>"created by main.main"}]
[13] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116399251, {"time"=>"2021-12-22T02:05:41.116399251Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  foo.go:5 +0x58"}]
[14] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138743.404566405, {"log"=>"2021-12-22T02:05:41.116402724Z stdout F "}]
[15] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116406309, {"time"=>"2021-12-22T02:05:41.116406309Z", "stream"=>"stdout", "_p"=>"F", "log"=>"goroutine 1 [chan receive]:"}]
[16] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116409950, {"time"=>"2021-12-22T02:05:41.11640995Z", "stream"=>"stdout", "_p"=>"F", "log"=>"runtime.gopark(0x4739b8, 0xc420024178, 0x46fcd7, 0xc, 0xc420028e17, 0x3)"}]
[17] kube.var.log.containers.test-log_default_count-e54975ff8223b51518793c17dd5a41c6c44e60acba86bef349637b0ab82bc5fd.log: [1640138741.116413696, {"time"=>"2021-12-22T02:05:41.116413696Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  /usr/local/go/src/runtime/proc.go:280 +0x12c fp=0xc420053e30 sp=0xc420053e00 pc=0x42503c"}]

To Reproduce
Run fluent-bit with above configuration

Expected behavior
both two multiline messages should be handled correctly

Screenshots

Your Environment

  • Version used:1.8.10
  • Configuration:see above
  • Environment name and version (e.g. Kubernetes? What version?):kubernetes 1.21
  • Operating System and version:ubuntu
  • Filters and plugins: tail, filter
@nokute78
Copy link
Collaborator

It is fixed on master, but it isn't released yet.
Issue: #4034
Patch: #4375

I tested it on master.

[INPUT]
        Name              tail
        Tag               kube.*
        Path              a.log
        multiline.parser  docker, cri
        Read_From_Head    true

[FILTER]
        Name                   multiline
        Match                  *
        multiline.key_content  log
        multiline.parser       go, java

[OUTPUT]
        Name stdout

a.log:

2021-12-22T02:36:56.941050368Z stdout F single line...
2021-12-22T02:36:56.941096311Z stdout F Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
2021-12-22T02:36:56.941101802Z stdout F     at com.myproject.module.MyProject.badMethod(MyProject.java:22)
2021-12-22T02:36:56.9411043Z stdout F     at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
2021-12-22T02:36:56.94110682Z stdout F     at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
2021-12-22T02:36:56.941109302Z stdout F     at com.myproject.module.MyProject.someMethod(MyProject.java:10)
2021-12-22T02:36:56.94111173Z stdout F     at com.myproject.module.MyProject.main(MyProject.java:6)
2021-12-22T02:36:56.941114604Z stdout F another line...
2021-12-22T02:36:56.941116989Z stdout F panic: my panic
2021-12-22T02:36:56.94111914Z stdout F 
2021-12-22T02:36:56.941122703Z stdout F goroutine 4 [running]:
2021-12-22T02:36:56.941125769Z stdout F panic(0x45cb40, 0x47ad70)
2021-12-22T02:36:56.941128769Z stdout F   /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
2021-12-22T02:36:56.941131055Z stdout F main.main.func1(0xc420024120)
2021-12-22T02:36:56.941133533Z stdout F   foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
2021-12-22T02:36:56.941135853Z stdout F runtime.goexit()
2021-12-22T02:36:56.941138281Z stdout F   /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
2021-12-22T02:36:56.941140724Z stdout F created by main.main
2021-12-22T02:36:56.941142941Z stdout F   foo.go:5 +0x58
2021-12-22T02:36:56.941145054Z stdout F

Output:

$ ../bin/fluent-bit -c a.conf 
Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2021/12/28 08:39:29] [ info] [engine] started (pid=11343)
[2021/12/28 08:39:29] [ info] [storage] version=1.1.5, initializing...
[2021/12/28 08:39:29] [ info] [storage] in-memory
[2021/12/28 08:39:29] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/12/28 08:39:29] [ info] [cmetrics] version=0.2.2
[2021/12/28 08:39:29] [ info] [input:tail:tail.0] multiline core started
[2021/12/28 08:39:29] [ info] [sp] stream processor started
[2021/12/28 08:39:29] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1453992 watch_fd=1 name=a.log
[0] kube.a.log: [1640140616.941050368, {"time"=>"2021-12-22T02:36:56.941050368Z", "stream"=>"stdout", "_p"=>"F", "log"=>"single line..."}]
[1] kube.a.log: [1640140616.941096311, {"time"=>"2021-12-22T02:36:56.941096311Z", "stream"=>"stdout", "_p"=>"F", "log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!
    at com.myproject.module.MyProject.badMethod(MyProject.java:22)
    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)
    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)
    at com.myproject.module.MyProject.someMethod(MyProject.java:10)
    at com.myproject.module.MyProject.main(MyProject.java:6)"}]
[2] kube.a.log: [1640140616.941114604, {"time"=>"2021-12-22T02:36:56.941114604Z", "stream"=>"stdout", "_p"=>"F", "log"=>"another line..."}]
[3] kube.a.log: [1640140616.941116989, {"time"=>"2021-12-22T02:36:56.941116989Z", "stream"=>"stdout", "_p"=>"F", "log"=>"panic: my panic

goroutine 4 [running]:
panic(0x45cb40, 0x47ad70)
  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c
main.main.func1(0xc420024120)
  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1
created by main.main
  foo.go:5 +0x58

"}]
^C[2021/12/28 08:39:34] [engine] caught signal (SIGINT)
[2021/12/28 08:39:34] [ info] [input] pausing tail.0
[2021/12/28 08:39:34] [ warn] [engine] service will shutdown in max 5 seconds
[2021/12/28 08:39:35] [ info] [engine] service has stopped (0 pending tasks)
[2021/12/28 08:39:35] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=1453992 watch_fd=1

@lecaros lecaros added the fixed label Jan 4, 2022
@xiaoping8385
Copy link
Author

@lecaros does 1.8.12 contain this fix?

@lecaros
Copy link
Contributor

lecaros commented Feb 17, 2022

Hi @xiaoping8385, no, it's only available for the future 1.9 release.

@Antiarchitect
Copy link

Antiarchitect commented Mar 23, 2022 •
edited
Loading

Just tried 1.9.0 have no errors in fluent-bit log but logs stopped. Config is:

    [INPUT]
        name              tail
        tag               k8s
        path              /var/log/containers/*.log
        db                /var/log/containers-logs.db
        key               raw
        path_key          source
        multiline.parser  cri
        ignore_older      1d
        mem_buf_limit     256MB
        refresh_interval  10
        skip_long_lines   on

Moreover debug logs full of:

[2022/03/23 13:52:42] [debug] [input chunk] skip ingesting data with 0 bytes

@xiaoping8385
Copy link
Author

@lecaros is that fixed in v1.9.0? cause I just tested and still not work, the same configuration as yours, but the result is like
kubectl logs fluent-bit-bnfsh -n pks-system -c fluent-bit
Fluent Bit v1.9.0

  • Copyright (C) 2015-2021 The Fluent Bit Authors
  • Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  • https://fluentbit.io
[0] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468042960, {"time"=>"2022-03-24T12:57:11.46804296Z", "stream"=>"stdout", "_p"=>"F", "log"=>"single line..."}]
[1] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468085444, {"time"=>"2022-03-24T12:57:11.468085444Z", "stream"=>"stdout", "_p"=>"F", "log"=>"Dec 14 06:41:08 Exception in thread "main" java.lang.RuntimeException: Something has gone wrong, aborting!"}]
[2] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468085444, {"log"=>"    at com.myproject.module.MyProject.badMethod(MyProject.java:22)"}]
[3] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468085444, {"log"=>"    at com.myproject.module.MyProject.oneMoreMethod(MyProject.java:18)"}]
[4] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468085444, {"log"=>"    at com.myproject.module.MyProject.anotherMethod(MyProject.java:14)"}]
[5] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468085444, {"log"=>"    at com.myproject.module.MyProject.someMethod(MyProject.java:10)"}]
[6] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468085444, {"log"=>"    at com.myproject.module.MyProject.main(MyProject.java:6)"}]
[7] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468106506, {"time"=>"2022-03-24T12:57:11.468106506Z", "stream"=>"stdout", "_p"=>"F", "log"=>"another line..."}]
[8] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468109176, {"time"=>"2022-03-24T12:57:11.468109176Z", "stream"=>"stdout", "_p"=>"F", "log"=>"panic: my panic"}]
[9] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468111843, {"time"=>"2022-03-24T12:57:11.468111843Z", "stream"=>"stdout", "_p"=>"F", "log"=>" "}]
[10] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468123257, {"time"=>"2022-03-24T12:57:11.468123257Z", "stream"=>"stdout", "_p"=>"F", "log"=>"goroutine 4 [running]:"}]
[11] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468126737, {"time"=>"2022-03-24T12:57:11.468126737Z", "stream"=>"stdout", "_p"=>"F", "log"=>"panic(0x45cb40, 0x47ad70)"}]
[12] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468130680, {"time"=>"2022-03-24T12:57:11.46813068Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  /usr/local/go/src/runtime/panic.go:542 +0x46c fp=0xc42003f7b8 sp=0xc42003f710 pc=0x422f7c"}]
[13] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468133476, {"time"=>"2022-03-24T12:57:11.468133476Z", "stream"=>"stdout", "_p"=>"F", "log"=>"main.main.func1(0xc420024120)"}]
[14] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468136563, {"time"=>"2022-03-24T12:57:11.468136563Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  foo.go:6 +0x39 fp=0xc42003f7d8 sp=0xc42003f7b8 pc=0x451339"}]
[15] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468139260, {"time"=>"2022-03-24T12:57:11.46813926Z", "stream"=>"stdout", "_p"=>"F", "log"=>"runtime.goexit()"}]
[16] kube.var.log.containers.test-log_default_count-8012f0320bd00297506315493f7a8b354dc7bdbbc367243b69a422ac0fbcffe4.log: [1648126631.468142003, {"time"=>"2022-03-24T12:57:11.468142003Z", "stream"=>"stdout", "_p"=>"F", "log"=>"  /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003f7e0 sp=0xc42003f7d8 pc=0x44b4d1"}]

@lecaros
Copy link
Contributor

lecaros commented Mar 24, 2022

I've just tried the test case from #4508 (comment), and it doesn't work. @nokute78, can you please confirm whether it works for you?

@nokute78
Copy link
Collaborator

The test case from #4508 (comment) didn't work from a7414f0
It worked correctly on Previous commit .
The PR is #4583 .

Note: The unit test of multiline on current master works correctly.
I think it also means the changing ingestion way may affect it.

Cc @edsiper

@github-actions
Copy link
Contributor

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

@github-actions github-actions bot added the Stale label Jun 26, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2022

This issue was closed because it has been stalled for 5 days with no activity.

@github-actions github-actions bot closed this as completed Jul 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fixed Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Antiarchitect @lecaros @nokute78 @xiaoping8385