Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIGSEGV when loading a plugin #6572

Closed
MichaelMure opened this issue Aug 15, 2019 · 12 comments
Closed

SIGSEGV when loading a plugin #6572

MichaelMure opened this issue Aug 15, 2019 · 12 comments
Labels
kind/bug A bug in existing code (including security flaws) not our bug

Comments

@MichaelMure
Copy link
Contributor

Version information:

go-ipfs version: 0.4.22-
Repo version: 7
System version: amd64/linux
Golang version: go1.12.8

Binaries are compiled locally.

Description:

I'm building go-ipfs and 3 plugins in a docker container. When starting go-ipfs, it crashes with the following stacktrace:

$ ipfs version
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xffffffffffffffff pc=0xffffffffffffffff]

runtime stack:
runtime.throw(0x17f0f5c, 0x2a)
        /usr/local/go/src/runtime/panic.go:617 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:374 +0x4a9

goroutine 1 [syscall]:
runtime.cgocall(0x14cf8d0, 0xc000355240, 0xc000010030)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000355210 sp=0xc0003551d8 pc=0x97267b
plugin._Cfunc_pluginOpen(0xc00048c000, 0xc000010030, 0x0)
        _cgo_gotypes.go:77 +0x4e fp=0xc000355240 sp=0xc000355210 pc=0x126ef5e
plugin.open.func1(0xc00048c000, 0x1001, 0x1001, 0xc000010030, 0x24d5880)
        /usr/local/go/src/plugin/plugin_dlopen.go:64 +0xb5 fp=0xc000355278 sp=0xc000355240 pc=0x1270145
plugin.open(0xc000042090, 0x27, 0x9a6aff, 0xc000048000, 0xc000355570)
        /usr/local/go/src/plugin/plugin_dlopen.go:64 +0x326 fp=0xc000355518 sp=0xc000355278 pc=0x126f396
plugin.Open(...)
        /usr/local/go/src/plugin/plugin.go:32
github.com/ipfs/go-ipfs/plugin/loader.loadPlugin(0xc000042090, 0x27, 0xc0003555e8, 0x9a6aff, 0xc000048000, 0xc0003555f8, 0x9a61fc)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/plugin/loader/load_linux.go:53 +0x40 fp=0xc000355598 sp=0xc000355518 pc=0x13b3940
github.com/ipfs/go-ipfs/plugin/loader.linuxLoadFunc.func1(0xc000042090, 0x27, 0x1acdf00, 0xc000470680, 0x0, 0x0, 0xa5304a, 0xc000470680)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/plugin/loader/load_linux.go:41 +0x2c4 fp=0xc000355678 sp=0xc000355598 pc=0x13b5254
path/filepath.walk(0xc000042090, 0x27, 0x1acdf00, 0xc000470680, 0xc00000e2c0, 0x0, 0x0)
        /usr/local/go/src/path/filepath/path.go:358 +0x426 fp=0xc000355750 sp=0xc000355678 pc=0xb05ee6
path/filepath.walk(0xc000040080, 0x12, 0x1acdf00, 0xc0004705b0, 0xc00000e2c0, 0x0, 0x20)
        /usr/local/go/src/path/filepath/path.go:382 +0x300 fp=0xc000355828 sp=0xc000355750 pc=0xb05dc0
path/filepath.Walk(0xc000040080, 0x12, 0xc00000e2c0, 0xc0004704e0, 0x0)
        /usr/local/go/src/path/filepath/path.go:404 +0xff fp=0xc000355888 sp=0xc000355828 pc=0xb0601f
github.com/ipfs/go-ipfs/plugin/loader.linuxLoadFunc(0xc000040080, 0x12, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/plugin/loader/load_linux.go:22 +0x91 fp=0xc0003558c8 sp=0xc000355888 pc=0x13b3881
github.com/ipfs/go-ipfs/plugin/loader.loadDynamicPlugins(0xc000040080, 0x12, 0x17c353d, 0x8, 0xc000355b88, 0x8, 0x0)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/plugin/loader/loader.go:72 +0xf1 fp=0xc000355920 sp=0xc0003558c8 pc=0x13b43e1
github.com/ipfs/go-ipfs/plugin/loader.NewPluginLoader(0xc000040080, 0x12, 0x1, 0x0, 0x0)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/plugin/loader/loader.go:37 +0x37d fp=0xc000355bf0 sp=0xc000355920 pc=0x13b3f2d
main.loadPlugins(0xc00004004a, 0xa, 0x11, 0xc0000700f0, 0x1)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/cmd/ipfs/main.go:61 +0x118 fp=0xc000355cd0 sp=0xc000355bf0 pc=0x14c9e38
main.mainRet.func2(0x1ac5480, 0xc000082540, 0xc000482000, 0x1a9b601, 0xc0000c2008, 0x1a99c00, 0xc00012c420)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/cmd/ipfs/main.go:141 +0x10d fp=0xc000355d38 sp=0xc000355cd0 pc=0x14cd71d
github.com/ipfs/go-ipfs-cmds/cli.Run(0x1ac5480, 0xc000082500, 0x24b71c0, 0xc0000b4020, 0x2, 0x2, 0xc0000c2000, 0xc0000c2008, 0xc0000c2010, 0x1946de0, ...)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/cli/run.go:107 +0x5bd fp=0xc000355ec8 sp=0xc000355d38 pc=0x14c121d
main.mainRet(0x0)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/cmd/ipfs/main.go:177 +0x421 fp=0xc000355f80 sp=0xc000355ec8 pc=0x14ca601
main.main()
        /root/go/pkg/mod/github.com/ipfs/[email protected]/cmd/ipfs/main.go:83 +0x22 fp=0xc000355f98 sp=0xc000355f80 pc=0x14ca1c2
runtime.main()
        /usr/local/go/src/runtime/proc.go:200 +0x20c fp=0xc000355fe0 sp=0xc000355f98 pc=0x99f6fc
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000355fe8 sp=0xc000355fe0 pc=0x9cd621

goroutine 21 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000c8b90)
        /root/go/pkg/mod/[email protected]/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
        /root/go/pkg/mod/[email protected]/stats/view/worker.go:32 +0x57

goroutine 34 [syscall]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 50 [select]:
github.com/ipfs/go-log/writer.(*MirrorWriter).logRoutine(0xc000150210)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/writer/writer.go:76 +0x11c
created by github.com/ipfs/go-log/writer.NewMirrorWriter
        /root/go/pkg/mod/github.com/ipfs/[email protected]/writer/writer.go:41 +0xb9

goroutine 6 [select]:
github.com/ipfs/go-ipfs/cmd/ipfs/util.(*IntrHandler).Handle.func1(0xc00000e280, 0xc000086060, 0xc000070070)
        /root/go/pkg/mod/github.com/ipfs/[email protected]/cmd/ipfs/util/signal.go:47 +0x13c
created by github.com/ipfs/go-ipfs/cmd/ipfs/util.(*IntrHandler).Handle
        /root/go/pkg/mod/github.com/ipfs/[email protected]/cmd/ipfs/util/signal.go:41 +0xbe

The problem doesn't seem to be related to any plugin in particular, as soon as one of them is present in $IPFS_PATH, this crash happen. To make things more complicated, this seems to only happen inside the docker container: if I extract the binaries (go-ipfs+plugins), it loads just fine.

Note: this whole thing was working fine ~20 days ago with go-ipfs v0.4.21. Alas, I can't run that anymore as the building process seems to be broken:

Building with GOPROXY='https://proxy.golang.org' GOPATH= go get github.com/ipfs/go-ipfs/cmd/[email protected] leads to missing package in the proxy (go: github.com/go-critic/[email protected]: unexpected status (https://proxy.golang.org/github.com/go-critic/go-critic/@v/v0.0.0-20181204210945-ee9bf5809ead.info): 410 Gone as an example).

Building with GOPATH= go get github.com/ipfs/go-ipfs/cmd/[email protected] leads to the following error: go: github.com/dgraph-io/[email protected]+incompatible: go.mod has post-v2 module path "github.com/dgraph-io/badger/v2" at revision v2.0.0-rc.2

@MichaelMure MichaelMure added the kind/bug A bug in existing code (including security flaws) label Aug 15, 2019
@MichaelMure
Copy link
Contributor Author

More on this, it seems that it's the ipfs/go-ipfs:v0.4.22 docker image that is somehow causing the problem.

What I'm doing in short is:

ARG GOIPFSVERSION=v0.4.22

# Build in a build container with golang
FROM golang:1.12-stretch AS builder

# ... build go-ipfs@${GOIPFSVERSION} and the plugins accordingly

# Now we override the official go-ipfs container with our things
FROM ipfs/go-ipfs:${GOIPFSVERSION}

# overwrite with our go-ipfs version
COPY --from=builder /root/go/bin/ipfs /usr/local/bin/ipfs

# ... copy the plugins 

# override the entrypoint with our script to copy the plugin binary
# before launching the normal /usr/local/bin/start_ipfs
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/docker_wrapper.sh"]
CMD ["daemon"]

While building for v0.4.22, if I change the second base image to ipfs/go-ipfs:v0.4.21 instead of ipfs/go-ipfs:v0.4.22, it solves the problem.

@MichaelMure
Copy link
Contributor Author

The base image for the running stage of ipfs/go-ipfs, busybox:1-glibc has been updated about a month ago. Most likely, it has now a different glibc version than the builder stage, golang:1.12-stretch.

@Stebalien
Copy link
Member

Is it possible to build within the new IPFS container?

@Kubuxu
Copy link
Member

Kubuxu commented Aug 15, 2019

Check ldd plugin.so for all plugins.

@Stebalien
Copy link
Member

If you're looking for a quick workaround:

  1. Download the IPFS source.
  2. Run go get github.com/ipfs/[email protected]
  3. Run echo 's3ds github.com/ipfs/go-ds-s3/plugin 0' >> plugin/loader/preload_list
  4. Run make build.

This will preload the plugin and build it into go-ipfs.

@Stebalien
Copy link
Member

Note: There's some documentation in docs/plugin.md.

@MichaelMure
Copy link
Contributor Author

Ha, nice trick. I'm good for now with reverting to ipfs/go-ipfs:v0.4.21 as my runner base image though.

Is it possible to build within the new IPFS container?

Do you mean building go-ipfs + plugins directly in the ipfs/go-ipfs container ? That would probably works indeed, but the real problem is that you have I think a glibc mismatch (depending on the actual version pulled by docker as it's not an explicit versioning) between the builder and runner base image of ipfs/go-ipfs. It doesn't seems to be a problem to run go-ipfs, but it's one to load a plugin.

@Stebalien
Copy link
Member

Do you mean building go-ipfs + plugins directly in the ipfs/go-ipfs container ?

Yes.

I'm good for now with reverting to ipfs/go-ipfs:v0.4.21 as my runner base image though.

Just make sure you're using 0.4.22.


The only issue I've found about this is with static builds: golang/go#13470

@MichaelMure
Copy link
Contributor Author

Sorry to be annoying, but imho it's really a bug in the Dockerfile (specifically here: https://github.com/ipfs/go-ipfs/blob/master/Dockerfile#L40) because the busybox image version is not set explicitly. As a consequence (and now by default), 1.31.0-glibc is pulled instead of as previously (I think) 1.30.1-glibc.

This has been a headache for me, but it might cause some subtle problem for the normal usage of the ipfs/go-ipfs container.

image

https://hub.docker.com/_/busybox?tab=tags

@Stebalien
Copy link
Member

Ah, got it. So the issue is that the busybox image is buster based while we're currently stretch based.

@Stebalien
Copy link
Member

(note: the glibc version shouldn't matter, as far as I can tell).

@MichaelMure
Copy link
Contributor Author

Dealt with by #6582

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws) not our bug
Projects
None yet
Development

No branches or pull requests

3 participants