distsql: panic of TPC-H query 3 when running on 6-node omega

[asubiotto]

Running TPC-H Query 3:

SELECT
    l_orderkey,
    SUM(l_extendedprice * (1 - l_discount)) AS revenue,
    o_orderdate,
    o_shippriority
FROM
    customer,
    orders,
    lineitem
WHERE
    c_mktsegment = 'MACHINERY'
    AND c_custkey = o_custkey
    AND l_orderkey = o_orderkey
    AND o_orderDATE < DATE '1995-03-10'
    AND l_shipdate > DATE '1995-03-10'
GROUP BY
    l_orderkey,
    o_orderdate,
    o_shippriority
ORDER BY
    revenue DESC,
    o_orderdate;

on 6-node omega with sha 78c5f93 results in:

E170324 14:13:25.364501 3713 sql/distsqlrun/server.go:216  [n1] [n1] communication error: rpc error: code = Canceled desc = context canceled
E170324 14:13:25.364788 3992 sql/distsqlrun/server.go:216  [n1] [n1] communication error: rpc error: code = Canceled desc = context canceled
panic: close of closed channel

goroutine 4098 [running]:
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*RowChannel).ProducerDone(0xc42390a630)
        /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:320 +0x2f
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*procOutputHelper).close(0xc4226d25f8)
        /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:280 +0x34
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*joinReader).Run(0xc4226d0a00, 0x7f326ac11040, 0xc421241e30, 0xc422a464a8)
        /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/joinreader.go:199 +0x1bf
created by github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Start
        /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:309 +0x337

This is caused by the same undesired behavior seen in #13989 but this issue tracks the panic caused by it. The execution plan can be found here.

The same thing does not happen on a local 3-node cluster (execution plan). I assume it is because the JoinReaders read from two different nodes rather than one, as is the case with omega.

[andreimatei]

So you think https://reviewable.io/reviews/cockroachdb/cockroach/14236#- fixes this?

…

On Fri, Mar 24, 2017 at 10:57 AM, Alfonso Subiotto Marqués < ***@***.***> wrote: Running TPC-H Query 3: SELECT l_orderkey, SUM(l_extendedprice * (1 - l_discount)) AS revenue, o_orderdate, o_shippriority FROM customer, orders, lineitem WHERE c_mktsegment = 'MACHINERY' AND c_custkey = o_custkey AND l_orderkey = o_orderkey AND o_orderDATE < DATE '1995-03-10' AND l_shipdate > DATE '1995-03-10' GROUP BY l_orderkey, o_orderdate, o_shippriority ORDER BY revenue DESC, o_orderdate; on 6-node omega with sha 78c5f93 results in: E170324 14:13:25.364501 3713 sql/distsqlrun/server.go:216 [n1] [n1] communication error: rpc error: code = Canceled desc = context canceled E170324 14:13:25.364788 3992 sql/distsqlrun/server.go:216 [n1] [n1] communication error: rpc error: code = Canceled desc = context canceled panic: close of closed channel goroutine 4098 [running]:github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*RowChannel).ProducerDone(0xc42390a630) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/base.go:320 +0x2fgithub.jparrowsec.cn/cockroachdb/cockroach/pkg/sql/distsqlrun.(*procOutputHelper).close(0xc4226d25f8) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:280 +0x34github.jparrowsec.cn/cockroachdb/cockroach/pkg/sql/distsqlrun.(*joinReader).Run(0xc4226d0a00, 0x7f326ac11040, 0xc421241e30, 0xc422a464a8) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/joinreader.go:199 +0x1bf created by github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Start /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:309 +0x337 This is caused by the same undesired behavior seen in #13989 <#13989> but this issue tracks the panic caused by it. The execution plan can be found here <https://raduberinde.github.io/decode.html?eJzclN9v2jAQx9_3V1j30rK5Ek4CdJYqGbVszTRgAvYwTVGV4htECjGyHWlVxf8-YbYSUmTW0UrV3i62P3ff-5G7h0JJHKQLNMC_AwMKESQUllpN0Ril18ebR7H8CbxJISuWpV0fJxSmSiPwe7CZzRE4fFJZMcJUogYKEm2a5c7vUmeLVN8JpSVqAxSGpeVEMCoCKlpUnEOyoqBK-8f1ij5f0DwrMLO4qIZtU9H565jbUGXhMkC5E8mRh57sEX6dmvlafF34cEBy_GFPBWtc6Gw2d9aD-oCKkIrI1W1PGlv3t3dknpr5rm_BIFm9_lRHWEjUnIg2ER3ylpwyzvlV7zLudz-TMyLOG0RERLTcg7VxzAB1ZzONs9SqmriHClOIr3qDyW9t46_9UxE0tqdhxY4qdqtityt2p_GMo_fy-RxR2bHStt5yEZxREb7zumV-t5P0NkffLz8tjVULd_shy62bpA65ICf97uV1POiNvp1wzseTUTz4WF0Lx_9M_6pc3Sj5aD02X6RI-Y2Re5aifx-y_35JHCjnK14Se9SO0CxVYbDelx1_CQWUM9y01KhST_GLVlPnfPM5dK_dgURjN7fB5iMu3BVby6rC7AlwUIeDOsyqMGseQ4d-OPTqjnbgZh2OvHDLX7GWF277I7e9MAv8dOcp9X4k3E-HfvjcK7zp1_3en7UfZv75Zodw_4TXm52s3vwKAAD__2AOoVo=> . The same thing does not happen on a local 3-node cluster (execution plan) <https://raduberinde.github.io/decode.html?eJzUlF9r2zwUxu_fTyHOTZN3KsR2_k1QUGiz1WNJRpJdjGGKZ525BscKkgwbJd99WN6axGFaE8Ngd5KOfud59FjyExRS4DzeoAb2GTyg4AOFACIKWyUT1FqqqlRvDMU3YD0KWbEtTbUcUUikQmBPYDKTIzBYx19yXGIsUAEFgSbOctt8q7JNrL7zpNRGbmz1TZYbVIzwEbkhV7PJ7X04ny4_XTHGVutlOH8LFBalYYR7lI8g2lGQpfmlvaMvd_VOZoXLlFQClT6U8ykfUD5uI_rbKPIHLXieFZgZ3OxFX37CvUZZWOsojhSi3QUZnBqifHhO7n92RS8yfh_rx8p80_hiTnL8ajrc696oLH20o2f3PuUB5X37Gf_JYyyxEPZ1DKsH8j_peIyxu-ltOJu8J9eEj7uE9wkf2A3VoM1dnaSpwjQ2smHuOT0K4d10vv7pbfVx1uF-d78aHIz73TZOVlKZZkTcv6Y8eOVs6_3Nx-hfKCYfpDj52fTaxLVEvZWFxuYVPeoXUUCRYn27tSxVgh-UTGzzerqwu-2CQG3q6qCehIUteZWtQ9hzwn037DvhwA0HZyj7Tbh_xplP4IETHh7BvSY8dMIjNzxywl7PTY_bxP3aLd1Qjnb__QgAAP__K9O2Jw==>. I assume it is because the JoinReaders read from two different nodes rather than one, as is the case with omega. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#14360>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXBcQOMWTr2cR3t9b8wWOAXXO5y286hks5ro9nzgaJpZM4MoXTm> .

[asubiotto]

I think it does.