Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segfault due to race in ebpf initialisation #2687

Closed
rade opened this issue Jul 6, 2017 · 2 comments · Fixed by weaveworks/tcptracer-bpf#45
Closed

segfault due to race in ebpf initialisation #2687

rade opened this issue Jul 6, 2017 · 2 comments · Fixed by weaveworks/tcptracer-bpf#45
Assignees
Labels
bug Broken end user or developer functionality; not working as the developers intended it

Comments

@rade
Copy link
Member

rade commented Jul 6, 2017

Apply

diff --git a/probe/endpoint/ebpf.go b/probe/endpoint/ebpf.go
index ff23b4e2..f9d470b5 100644
--- a/probe/endpoint/ebpf.go
+++ b/probe/endpoint/ebpf.go
@@ -7,6 +7,7 @@ import (
        "strconv"
        "sync"
        "syscall"
+       "time"
 
        log "github.com/Sirupsen/logrus"
        "github.com/weaveworks/common/fs"
@@ -96,6 +97,9 @@ func newEbpfTracker() (eventTracker, error) {
                tracer:          t,
        }
 
+       log.Infof("delaying setting of ebpfTracker")
+       time.Sleep(5 * time.Second)
+
        ebpfTracker = tracker
        return tracker, nil
 }

and then start a probe

$ sudo prog/scope --mode=probe --weave=false
INFO[0000] publishing to: 127.0.0.1:4040                
<probe> INFO: 2017/07/06 14:06:01.763599 command line args: --mode=probe --weave=false
<probe> INFO: 2017/07/06 14:06:01.763676 probe starting, version 0dbea45e, ID 3f9c8145f01926a1
<probe> ERRO: 2017/07/06 14:06:01.766805 Error fetching app details: Get http://127.0.0.1:4040/api: dial tcp 127.0.0.1:4040: getsockopt: connection refused
<probe> INFO: 2017/07/06 14:06:02.267125 delaying setting of ebpfTracker
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x176e70a]

goroutine 73 [running]:
github.com/weaveworks/scope/probe/endpoint.(*EbpfTracker).handleConnection(0x0, 0xc400000003, 0xc4219699f0, 0xc, 0xc421969a10, 0xd, 0x1bba04a, 0x5315, 0xc421969a30, 0xa)
	/go/src/github.com/weaveworks/scope/probe/endpoint/ebpf.go:208 +0x3a
github.com/weaveworks/scope/probe/endpoint.tcpEventCbV4(0x94fc54310ad1, 0x3, 0x531500000003, 0xc421969978, 0x5, 0xc421969990, 0x10, 0x10, 0xc4219699b0, 0x10, ...)
	/go/src/github.com/weaveworks/scope/probe/endpoint/ebpf.go:126 +0x217
github.com/weaveworks/scope/vendor/github.com/weaveworks/tcptracer-bpf/pkg/tracer.NewTracer.func1(0xc422e689c0, 0xc420468240, 0x287bf60, 0xc420468300, 0x287bf58)
	/go/src/github.com/weaveworks/scope/vendor/github.com/weaveworks/tcptracer-bpf/pkg/tracer/tracer.go:84 +0x1d6
created by github.com/weaveworks/scope/vendor/github.com/weaveworks/tcptracer-bpf/pkg/tracer.NewTracer
	/go/src/github.com/weaveworks/scope/vendor/github.com/weaveworks/tcptracer-bpf/pkg/tracer/tracer.go:89 +0x6d3

That's because the tracer invokes callback functions that dispatch via a global(!) var which we can only set after initialising the tracer. Chicken and egg. Easy to fix.

@rade rade added the bug Broken end user or developer functionality; not working as the developers intended it label Jul 6, 2017
@rade rade self-assigned this Jul 6, 2017
@2opremio
Copy link
Contributor

2opremio commented Jul 6, 2017

CC @alban

@rade
Copy link
Member Author

rade commented Jul 6, 2017

I'm on it, @fons.

rade added a commit that referenced this issue Jul 7, 2017
We defer starting the ebpf tracer until we've set the global var which
is referenced by the callback functions. Previously the var could be
unset when the callbacks are invoked, resulting in a segfault.

Fixes #2687.
rade added a commit that referenced this issue Jul 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Broken end user or developer functionality; not working as the developers intended it
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants