Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ipfs commands panic when HTTP_PROXY set #4998

Closed
BlinkyStitt opened this issue May 4, 2018 · 2 comments · Fixed by ipfs/go-ipfs-cmds#108
Closed

ipfs commands panic when HTTP_PROXY set #4998

BlinkyStitt opened this issue May 4, 2018 · 2 comments · Fixed by ipfs/go-ipfs-cmds#108
Assignees
@Stebalien
Labels
kind/bug A bug in existing code (including security flaws)

Comments

@BlinkyStitt
Copy link

BlinkyStitt commented May 4, 2018 •
edited
Loading

Version information:

$ ipfs version --all
go-ipfs version: 0.4.14-5db3846
Repo version: 6
System version: amd64/linux
Golang version: go1.9.4

(also fails under 0.4.13 and 0.4.15-rc1)

Type:

Bug

Description:

A few weeks ago I setup a Gitlab job that runs ipfs add and then updates my DNS server. This was working great 2 weeks ago, but when I went to update a site today, I got this error:

Running with gitlab-runner 10.6.0 (a3543a27)
  on leafblower 273e0857
Using Docker executor with image ipfs/go-ipfs:v0.4.14 ...
Pulling docker image ipfs/go-ipfs:v0.4.14 ...
Using docker image sha256:318eb01cab621e5fa4a1f40dfbdbe822c04e4c1a22b36dbb13400d1c34a454ce for ipfs/go-ipfs:v0.4.14 ...
Running on runner-273e0857-project-84-concurrent-0 via 29e774d0b471...
Fetching changes...
Removing ipfs_hash
HEAD is now at d8da9f2 do that in ipfs-dnslink
From https://gitlab.stytt.com/static/bryan.stitthappens.com
   d8da9f2..15d23b3  master     -> origin/master
Checking out 15d23b36 as master...
Skipping Git submodules setup
$ echo "IPFS_API is $IPFS_API"
IPFS_API is /ip4/10.11.12.130/tcp/5001
$ ipfs version --all
go-ipfs version: 0.4.14-5db3846
Repo version: 6
System version: amd64/linux
Golang version: go1.9.4
$ ipfs --api="$IPFS_API" add --pin=false ./site/index.html
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x11ec2e4]
 
goroutine 22 [running]:
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http.parseResponse(0xc420126cf0, 0xc4204daaf0, 0xc420126cf0, 0x0, 0x0, 0xc420022de0)
    /go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http/parse.go:215 +0x304
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http.(*client).Send(0xc4204fcf00, 0xc4204daaf0, 0xc4204dabd0, 0x1d55340, 0xc42044b200, 0x1)
    /go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http/client.go:178 +0x515
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http.(*client).Execute(0xc4204fcf00, 0xc4204daaf0, 0x1d55340, 0xc42044b200, 0x1d466c0, 0xc4204fcec0, 0x0, 0x0)
    /go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http/client.go:108 +0xde
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli.Run.func4(0x7fa3ce991030, 0xc4204fcf00, 0xc4204daaf0, 0x1d55380, 0xc4204dabd0, 0x1d466c0, 0xc4204fcec0, 0xc420059440)
    /go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli/run.go:146 +0x70
created by gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli.Run
    /go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli/run.go:145 +0x6ed
ERROR: Job failed: exit code 2

Under 0.4.13, the error is different:

$ ipfs version --all
go-ipfs version: 0.4.13-cc01b7f
Repo version: 6
System version: amd64/linux
Golang version: go1.9.2
$ ipfs --api="$IPFS_API" add --hidden --pin --quieter --recursive ./site >ipfs_hash
goroutine 1 [running]:
runtime/debug.Stack(0x8, 0x8, 0xc42000e758)
/usr/local/go/src/runtime/debug/stack.go:24 +0xa7
runtime/debug.PrintStack()
/usr/local/go/src/runtime/debug/stack.go:16 +0x22
gx/ipfs/QmSU6eubNdhXjFBJBSksTp8kv8YRub8mGAPv8tVJHmL2EU/go-ipfs-util.ErrCast(0xc4204d1030, 0x12c31e0)
/go/src/gx/ipfs/QmSU6eubNdhXjFBJBSksTp8kv8YRub8mGAPv8tVJHmL2EU/go-ipfs-util/util.go:37 +0x20
github.com/ipfs/go-ipfs/core/commands.glob..func7(0x1c70760, 0xc4200fe180, 0x1c6f9e0, 0xc4204d1030)
/go/src/github.com/ipfs/go-ipfs/core/commands/add.go:339 +0xd75
main.callCommand(0x1c675a0, 0xc4204fcd80, 0x1c70760, 0xc4200fe180, 0x1d3e2a0, 0x1c3d680, 0x0, 0x0, 0x11b9dcd, 0xc420157e90)
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:360 +0x4e3
main.(*cmdInvocation).Run(0xc4204fccc0, 0x1c675a0, 0xc4204fcd80, 0x1c5cae0, 0xc4201d0220, 0x1c675a0, 0xc4204fcd80)
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:192 +0x116
main.mainRet(0x0)
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:157 +0x3ca
main.main()
/go/src/github.com/ipfs/go-ipfs/cmd/ipfs/main.go:64 +0x22
Error: cast error

And under master:

Running with gitlab-runner 10.7.1 (b9bba623)
  on leafblower 273e0857
Using Docker executor with image ipfs/go-ipfs:master ...
Pulling docker image ipfs/go-ipfs:master ...
Using docker image sha256:8c48292e66e2fe5264e22072aefa9d3e06cc9e58d57e8a71b3ba64e6407d4ade for ipfs/go-ipfs:master ...
Running on runner-273e0857-project-84-concurrent-0 via 23de6de4c212...
Fetching changes...
HEAD is now at 15d23b3 try adding a single file
From https://gitlab.stytt.com/static/bryan.stitthappens.com
   15d23b3..dcd6ab0  master     -> origin/master
Checking out dcd6ab05 as master...
Skipping Git submodules setup
$ echo "IPFS_API is $IPFS_API"
IPFS_API is /ip4/10.11.12.130/tcp/5001
$ ipfs version --all
go-ipfs version: 0.4.15-rc1-
Repo version: 6
System version: amd64/linux
Golang version: go1.9.5
$ ipfs --api="$IPFS_API" add --pin=false ./site/index.html
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x11e7f94]

goroutine 22 [running]:
gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/http.parseResponse(0xc42012eea0, 0xc4204ecaf0, 0xc42012eea0, 0x0, 0x0, 0xc420022de0)
	/go/src/gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/http/parse.go:215 +0x304
gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/http.(*client).Send(0xc4204e9080, 0xc4204ecaf0, 0xc4204ecbd0, 0x1d40ce0, 0xc42044b1d0, 0x1)
	/go/src/gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/http/client.go:178 +0x515
gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/http.(*client).Execute(0xc4204e9080, 0xc4204ecaf0, 0x1d40ce0, 0xc42044b1d0, 0x1d33660, 0xc4204e9040, 0x0, 0x0)
	/go/src/gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/http/client.go:108 +0xde
gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/cli.Run.func4(0x7f6dddb61148, 0xc4204e9080, 0xc4204ecaf0, 0x1d40d20, 0xc4204ecbd0, 0x1d33660, 0xc4204e9040, 0xc4200594a0)
	/go/src/gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/cli/run.go:146 +0x70
created by gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/cli.Run
	/go/src/gx/ipfs/QmTjNRVt2fvaRFu93keEC7z5M1GS1iH6qZ9227htQioTUY/go-ipfs-cmds/cli/run.go:145 +0x6ed
ERROR: Job failed: exit code 2

If I use the docker container to ipfs add on leafblower, everything works fine. This probably means it is an issue with GitLab and not IPFS itself, so I've opened a support ticket with them, too.

This is the IPFS part of my .gitlab-ci.yml:

# https://gitlab.stytt.com/groups/static/-/settings/ci_cd
# - IPFS_API=/ip4/10.11.12.130/tcp/5001

stages:
- upload

ipfs add:
  stage: upload
  image:
    name: ipfs/go-ipfs:master
    entrypoint: [""]
  script:
    - echo "IPFS_API is $IPFS_API"
    - ipfs version --all
    # debugging...
    - ipfs --api="$IPFS_API" add --pin=false ./site/index.html
    - ipfs --api="$IPFS_API" add --hidden --recursive ./site
    # end debugging...
    - ipfs --api="$IPFS_API" add --hidden --pin --quieter --recursive ./site >ipfs_hash
    - cat ipfs_hash

Any ideas how to get more useful logs out of ipfs? I can run ipfs --api="$IPFS_API" add --hidden --pin --quieter --recursive ./site on my laptop just fine with and without docker. It's just under gitlab that it dies.

My previous successes were using gitlab-runner 10.6.0, so I rolled back to that version from 10.7, but that doesn't seem to have made any difference.
@BlinkyStitt BlinkyStitt changed the title ipfs add panics under Gitlab Runner ipfs commands panic under Gitlab Runner May 4, 2018
@BlinkyStitt
Copy link
Author

BlinkyStitt commented May 4, 2018 •
edited
Loading

I dug in more and ipfs --api="$IPFS_API" id also panics when run from inside a Gitlab runner (gitlab.stytt.com:5001/docker/ipfs is just ipfs/go-ipfs:v0.4.14 tagged and pushed into my personal repo):

Running with gitlab-runner 10.6.0 (a3543a27)
  on games d7f3ff28
Using Docker executor with image gitlab.stytt.com:5001/docker/ipfs:latest ...
Pulling docker image gitlab.stytt.com:5001/docker/ipfs:latest ...
Using docker image sha256:318eb01cab621e5fa4a1f40dfbdbe822c04e4c1a22b36dbb13400d1c34a454ce for gitlab.stytt.com:5001/docker/ipfs:latest ...
Running on runner-d7f3ff28-project-84-concurrent-0 via 754a7ada8e08...
Fetching changes...
HEAD is now at 06cc883 do a louder add
From https://gitlab.stytt.com/static/bryan.stitthappens.com
   06cc883..98c97db  master     -> origin/master
Checking out 98c97dbc as master...
Skipping Git submodules setup
$ echo "IPFS_API is $IPFS_API"
IPFS_API is /ip4/10.11.12.130/tcp/5001
$ ipfs version --all
go-ipfs version: 0.4.14-5db3846
Repo version: 6
System version: amd64/linux
Golang version: go1.9.4
$ ipfs --api="$IPFS_API" id
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x11ec2e4]

goroutine 13 [running]:
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http.parseResponse(0xc420252000, 0xc4201aeb60, 0xc420252000, 0x0, 0x0, 0x0)
	/go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http/parse.go:215 +0x304
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http.(*client).Send(0xc42018da40, 0xc4201aeb60, 0x14745e0, 0x4, 0xc4201907d8, 0x1)
	/go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http/client.go:178 +0x515
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http.(*client).Execute(0xc42018da40, 0xc4201aeb60, 0x1d55380, 0xc4201aec40, 0x1d466c0, 0xc42018da00, 0x0, 0x0)
	/go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/http/client.go:108 +0xde
gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli.Run.func4(0x7f77b704b8e0, 0xc42018da40, 0xc4201aeb60, 0x1d55380, 0xc4201aec40, 0x1d466c0, 0xc42018da00, 0xc42006d500)
	/go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli/run.go:146 +0x70
created by gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli.Run
	/go/src/gx/ipfs/QmfAkMSt9Fwzk48QDJecPcwCUjnf2uG7MLnmCGTp4C6ouL/go-ipfs-cmds/cli/run.go:145 +0x6ed

The same container works fine when run manually:

$ docker run --rm -it --env IPFS_API="/ip4/10.11.12.130/tcp/5001" --entrypoint sh gitlab.stytt.com:5001/docker/ipfs:latest 
/ # ipfs --api="$IPFS_API" id
{
	"ID": "QmTDzoCjEZqzhZbdpXSyYhJxpVDa7QNSyV3JG6qtLBe35S",
	"PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKeSoOMLMLCdqBMs0Gv8v68RGZrW44eVNKq+du6446OPoNXgX19xI3DpkVmmeHy315UdVjSDC87BUawsi7aFYEQbgYXIcGFtZWbJfGemzj2G23/0T6JjS/+aZyXgbpsAopLU0odh/x7nQB9xBJWPMVN7I1xRxCH0C/niUpxdvzKYsg3+5R4XiCvxXMH7Dz0h1hHSgcgXPP/095gvAo10x/+onl82/fYD8a1AHHl5y5Mf5NTMlp87oRJ63t16hiHT2r/v6SyCbxDHsP5NnX3ohpIq9z84A6UJp+xICciNUmkob6QPMnJVlUQC/DTo1ZtZB88oA2/iIgfXx3o5Gxb015AgMBAAE=",
	"Addresses": [
		"/ip4/127.0.0.1/tcp/4001/ipfs/QmTDzoCjEZqzhZbdpXSyYhJxpVDa7QNSyV3JG6qtLBe35S",
		"/ip4/10.11.12.130/tcp/4001/ipfs/QmTDzoCjEZqzhZbdpXSyYhJxpVDa7QNSyV3JG6qtLBe35S",
		"/ip4/192.168.0.14/tcp/4001/ipfs/QmTDzoCjEZqzhZbdpXSyYhJxpVDa7QNSyV3JG6qtLBe35S"
	],
	"AgentVersion": "go-ipfs/0.4.14/5db3846",
	"ProtocolVersion": "ipfs/0.1.0"
}

Stebalien added a commit to ipfs/go-ipfs-cmds that referenced this issue May 5, 2018
@Stebalien
check if we can decode an error before trying
feb32b8 
avoids a nil pointer dereference

fixes ipfs/kubo#4998
@Stebalien Stebalien mentioned this issue May 5, 2018
Merged
@ghost ghost assigned Stebalien May 5, 2018
@ghost ghost added the status/in-progress In progress label May 5, 2018
Stebalien added a commit to ipfs/go-ipfs-cmds that referenced this issue May 5, 2018
@Stebalien
test for ipfs/kubo#4998
d6d80d2
@Stebalien Stebalien added the kind/bug A bug in existing code (including security flaws) label May 5, 2018
@BlinkyStitt BlinkyStitt changed the title ipfs commands panic under Gitlab Runner ipfs commands panic when HTTP_PROXY set May 5, 2018
@BlinkyStitt
Copy link
Author

Did a little more digging and it looks like the cause isn't actually gitlab, but my gitlab runner's HTTP_PROXY setting. I added my ipfs server to NO_PROXY and my job is working again!

I think @Stebalien's PR should still be done to get us a better error message.

Stebalien added a commit to ipfs/go-ipfs-cmds that referenced this issue Jun 26, 2018
@Stebalien
test for ipfs/kubo#4998
4b7dc1a
Stebalien added a commit to ipfs/go-ipfs-cmds that referenced this issue Jun 26, 2018
@Stebalien
check if we can decode an error before trying
0b93697 
avoids a nil pointer dereference

fixes ipfs/kubo#4998
Stebalien added a commit to ipfs/go-ipfs-cmds that referenced this issue Jun 26, 2018
@Stebalien
test for ipfs/kubo#4998
b8222c5
@ghost ghost removed the status/in-progress In progress label Aug 10, 2018
@Stebalien Stebalien mentioned this issue Aug 16, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Stebalien Stebalien

Labels
kind/bug A bug in existing code (including security flaws)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

check if we can decode an error before trying ipfs/go-ipfs-cmds
2 participants
@Stebalien @BlinkyStitt