feat: introduce the framework of sqlsmith

[neverchanje]

What's changed and what's your intention?

The example sqlsmith-generated SQL:

SELECT TIMESTAMP '2022-06-10 23:20:03' AS col_0, SMALLINT '1' AS col_1, FLOAT '14.00766166396339' AS col_2, 56.34405973613994 AS col_3 FROM partsupp AS t0, region AS t1 WHERE t0.ps_supplycost <= (((76 - (37 % SMALLINT '92')) >> INT '53') << (SMALLINT '77' # ((SMALLINT '6' << SMALLINT '87') << (((SMALLINT '11' # SMALLINT '68') << (SMALLINT '1' - (SMALLINT '63' * INT '5'))) >> (t0.ps_suppkey >> (SMALLINT '48' % SMALLINT '41')))))) LIMIT 14
SELECT DATE '2022-06-13' + (INT '6' % (((((SMALLINT '27' # SMALLINT '32') | SMALLINT '22') >> SMALLINT '24') # SMALLINT '87') / SMALLINT '63')) AS col_0, TIMESTAMP '2022-06-17 09:10:38' AS col_1, REAL '67.34746316318869' AS col_2, (INTERVAL '798279' * ((t2.c_custkey << (SMALLINT '91' << INT '24')) >> SMALLINT '41')) < (t3.l_discount * INTERVAL '906922') AS col_3 FROM nation AS t0, lineitem AS t1, customer AS t2, lineitem AS t3, customer AS t4 WHERE false
SELECT t2.c_acctbal AS col_0, (INTERVAL '323373' / FLOAT '71.77997231857036') + ((((INTERVAL '118695' + TIMESTAMP '2022-06-10 12:07:08') + INTERVAL '900004') - INTERVAL '379835') - (FLOAT '66.82124471287459' * INTERVAL '761419')) AS col_1 FROM part AS t0, supplier AS t1, customer AS t2 LIMIT 47
SELECT REAL '36.66525673061021' + ((t0.l_linenumber << SMALLINT '65') / (round(22.502892250863237, INT '4') / SMALLINT '30')) AS col_0 FROM lineitem AS t0, lineitem AS t1 WHERE (SMALLINT '19' & SMALLINT '91') > (9 + FLOAT '50.10338423028955')
SELECT 45.92007126393105 AS col_0, (((t2.c_acctbal * ((((29.32164110038171 + REAL '66.15275368873897') + t1.p_retailprice) * round(38.99341338293072, t0.s_nationkey)) + 32)) + 67.46192340702622) + t4.c_acctbal) / (REAL '17.611260215308167' + (((SMALLINT '59' & SMALLINT '2') / (SMALLINT '21' * SMALLINT '31')) * SMALLINT '1')) AS col_1, 31 >> (SMALLINT '8' - SMALLINT '18') AS col_2 FROM supplier AS t0, part AS t1, customer AS t2, region AS t3, customer AS t4 WHERE false

Additionally, this PR implements a CLI tool sqlsmith so that we can test queries end to end. The typical panics are caused by the inconsistency of exprs between frontend and compute. Some unsupported exprs are actually passed to the computes.

cargo build
RUST_LOG=INFO ./target/debug/sqlsmith --testdata ./src/tests/sqlsmith/tests/testdata

Currently, this tool doesn't support randomizing the GROUP BY clause or JOIN, etc. Also it only runs against the optimizer. The plan is to fully construct more complex SELECT queries, CREATE MATERIALIZED VIEW queries.

Checklist

• I have written necessary docs and comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

#2571

[github-actions]

license-eye has totally checked 853 files.

Valid	Invalid	Ignored	Fixed
849	3	1	0

Click to see the invalid file list

• src/tests/sqlsmith/src/expr.rs
• src/tests/sqlsmith/src/lib.rs
• src/tests/sqlsmith/src/test_runner.rs

[codecov]

Codecov Report

Merging #3305 (eefb18a) into main (daf9222) will increase coverage by 0.05%.
The diff coverage is 78.26%.

@@            Coverage Diff             @@
##             main    #3305      +/-   ##
==========================================
+ Coverage   73.18%   73.23%   +0.05%     
==========================================
  Files         755      756       +1     
  Lines      102431   102476      +45     
==========================================
+ Hits        74963    75051      +88     
+ Misses      27468    27425      -43     

Flag	Coverage Δ
rust	73.23% <78.26%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/frontend/src/binder/mod.rs	100.00% <ø> (ø)
src/frontend/src/expr/mod.rs	87.09% <ø> (ø)
src/sqlparser/test_runner/src/lib.rs	64.78% <ø> (ø)
src/frontend/src/session.rs	41.46% <15.38%> (-0.48%)	⬇️
src/tests/sqlsmith/src/expr.rs	89.47% <87.50%> (ø)
src/sqlparser/src/ast/mod.rs	88.75% <92.85%> (+0.05%)	⬆️
src/frontend/src/expr/type_inference/mod.rs	95.45% <95.00%> (+0.71%)	⬆️
src/frontend/src/expr/type_inference/cast.rs	100.00% <100.00%> (ø)
src/frontend/src/expr/type_inference/func.rs	99.47% <100.00%> (+0.79%)	⬆️
... and 8 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[liurenjie1024]

Nice PR. I think the next step is to run sqls and compare it with postgresql?

[neverchanje]

Nice PR. I think the next step is to run sqls and compare it with postgresql?

No I personally do not want to use sqlsmith for verifying correctness. It was also stated in the blog written by cockroach team, https://www.cockroachlabs.com/blog/sqlsmith-randomized-sql-testing/

This kind of fuzz testing is not able to deduce correctness. Today, its goal is limited to finding panics, which it’s quite good at. Correctness is difficult because we don’t have any oracle of truth, which would require a known working SQL engine, which is exactly the thing we’re trying to break. We are unable to use Postgres as an oracle because CockroachDB has slightly different semantics and SQL support, and generating queries that execute identically on both is tricky and doesn’t allow us to use the full CockroachDB grammar.

For example, we can not test the correctness of HOP/TUMBLE against Postgres. It's unsupported there. Our goal is to find panics inside our system. At least to ensure no sql would cause any panic.

[liurenjie1024]

Generally LGTM.