Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flakes due to 'shared dir' in grpcproxy e2e tests #12965

Merged
merged 5 commits into from
May 15, 2021

Conversation

ptabor
Copy link
Contributor

@ptabor ptabor commented May 14, 2021

Multiple proxy instances were using the same directory in grpcproxy e2e tests, leading to test flakes.

In particular it manifested as:

/home/ptab/corp/etcd/bin/etcd-1483016: {"level":"warn","ts":"2021-05-13T22:50:21.219+0200","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"5ac8aa22f1eb4c8f","error":"failed to read 5ac8aa22f1eb4c8f on stream Message (unexpected EOF)"}
/home/ptab/corp/etcd/bin/etcd-1483016: {"level":"warn","ts":"2021-05-13T22:50:21.220+0200","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"72e86eb5e356a737","remote-peer-id":"5ac8aa22f1eb4c8f","error":"unexpected EOF"}
    ctl_v3_test.go:223: could not start etcd process cluster (Cannot start: match not found. Set EXPECT_DEBUG for more info Err: read /dev/ptmx: input/output error, last lines:
        {"level":"warn","ts":"2021-05-13T22:50:21.017+0200","caller":"transport/listener.go:249","msg":"automatically generate certificates","certificate-validity-bound-not-after":"2022-05-13T22:50:21.017+0200"}
        {"level":"info","ts":"2021-05-13T22:50:21.191+0200","caller":"transport/listener.go:298","msg":"created cert file","path":"/tmp/TestCtlV3MemberRemoveClientAutoTLS2632588310/001/default.proxy/fixtures/proxy/cert.pem"}
        {"level":"info","ts":"2021-05-13T22:50:21.192+0200","caller":"transport/listener.go:319","msg":"created key file","path":"/tmp/TestCtlV3MemberRemoveClientAutoTLS2632588310/001/default.proxy/fixtures/proxy/key.pem"}
        {"level":"info","ts":"2021-05-13T22:50:21.192+0200","caller":"etcdmain/grpc_proxy.go:201","msg":"gRPC proxy server TLS","tls-info":"cert = /tmp/TestCtlV3MemberRemoveClientAutoTLS2632588310/001/default.proxy/fixtures/proxy/cert.pem, key = /tmp/TestCtlV3MemberRemoveClientAutoTLS2632588310/001/default.proxy/fixtures/proxy/key.pem, client-cert=/tmp/TestCtlV3MemberRemoveClientAutoTLS2632588310/001/default.proxy/fixtures/proxy/cert.pem, client-key=/tmp/TestCtlV3MemberRemoveClientAutoTLS2632588310/001/default.proxy/fixtures/proxy/key.pem, trusted-ca = , client-cert-auth = false, crl-file = "}
        {"level":"fatal","ts":"2021-05-13T22:50:21.202+0200","caller":"etcdmain/grpc_proxy.go:381","msg":"failed to create TLS listener","error":"tls: private key does not match public key","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.mustListenCMux\n\t/home/ptab/corp/etcd/server/etcdmain/grpc_proxy.go:381\ngo.etcd.io/etcd/server/v3/etcdmain.startGRPCProxy\n\t/home/ptab/corp/etcd/server/etcdmain/grpc_proxy.go:203\ngithub.jparrowsec.cn/spf13/cobra.(*Command).execute\n\t/home/ptab/private/golang/pkg/mod/github.com/spf13/[email protected]/command.go:854\ngithub.jparrowsec.cn/spf13/cobra.(*Command).ExecuteC\n\t/home/ptab/private/golang/pkg/mod/github.com/spf13/[email protected]/command.go:958\ngithub.jparrowsec.cn/spf13/cobra.(*Command).Execute\n\t/home/ptab/private/golang/pkg/mod/github.com/spf13/[email protected]/command.go:895\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/home/ptab/corp/etcd/server/etcdmain/main.go:32\nmain.main\n\t/home/ptab/corp/etcd/server/main.go:32\nruntime.main\n\t/usr/lib/google-golang/src/runtime/proc.go:225"}
        )
--- FAIL: TestCtlV3MemberRemoveClientAutoTLS (2.93s)

@ptabor ptabor force-pushed the 20210513-flake-in-e2e-debugging branch 2 times, most recently from f4a99dd to 5c800ce Compare May 14, 2021 05:41
@ptabor ptabor force-pushed the 20210513-flake-in-e2e-debugging branch from 8d0489c to d8550de Compare May 14, 2021 20:46
@ptabor ptabor merged commit ddc4f47 into etcd-io:main May 15, 2021
@ptabor ptabor deleted the 20210513-flake-in-e2e-debugging branch May 15, 2021 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants