Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[multistage][test] add table expression tests #9817

Merged
merged 4 commits into from
Nov 28, 2022
Merged

[multistage][test] add table expression tests #9817

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f24ae62
add table expression tests
Nov 16, 2022
cd42e43
adding more tests per comments
Nov 17, 2022
73ed469
add more tests
Nov 17, 2022
36cab68
Merge branch 'master' into pr_add_table_tests
walterddr Nov 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ...untime/src/test/java/org/apache/pinot/query/runtime/queries/ResourceBasedQueriesTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ public class ResourceBasedQueriesTest extends QueryRunnerTestBase {
"SelectExpressions.json",
"ValueExpressions.json",
"NumericTypes.json",
"TableExpressions.json",
"CharacterTypes.json",
"BinaryTypes.json",
"TimeTypes.json",
Expand Down
110 changes: 110 additions & 0 deletions pinot-query-runtime/src/test/resources/queries/TableExpressions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
{
Copy link
Contributor

@siddharthteotia siddharthteotia Nov 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High level question on table expression tests?

By table expressions do you mean CTEs ? CTEs create named queries using WITH afaik.

May be we can call these Subqueries.json to distinguish from CTEs ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table expression as defined in postgresql 7.2 section. (e.g. all clauses that alters the behavior of a SQL after choosing the tables)
some of the sections we don't support is not included such as:

"where_clause_tests": {
"psql": "7.2.2",
"tables": {
"tbl": {
"schema": [
{"name": "strCol", "type": "STRING"},
{"name": "intCol", "type": "INT"}
],
"inputs": [
["foo", 1],
["bar", 2],
["alice", 42],
["bob", 196883]
]
}
},
"queries": [
{ "sql": "SELECT * FROM {tbl} WHERE intCol > 5" },
{ "sql": "SELECT * FROM {tbl} WHERE strCol IN ('foo', 'bar')" },
{ "sql": "SELECT * FROM {tbl} WHERE intCol IN (196883, 42)" },
{ "sql": "SELECT * FROM {tbl} WHERE intCol NOT IN (196883, 42) AND strCol IN ('alice')" },
{ "sql": "SELECT * FROM {tbl} WHERE strCol IN (SELECT strCol FROM {tbl} WHERE intCol > 100)" },
{ "sql": "SELECT * FROM {tbl} WHERE intCol < (SELECT SUM(intCol) FROM {tbl} AS b WHERE strCol BETWEEN 'bar' AND 'foo')" },
{ "sql": "SELECT * FROM {tbl} WHERE intCol BETWEEN 0 AND 100 AND strCol BETWEEN 'bar' AND 'foo'" },
{ "sql": "SELECT * FROM {tbl} WHERE intCol IN (SELECT a.intCol FROM {tbl} AS a JOIN {tbl} AS b ON a.strCol = b.strCol WHERE MOD(a.intCol, 2) = MOD(b.intCol, 2))" },
{
"ignored": true,
"comments": "Relation Decorrelator not supported",
"sql": "SELECT * FROM {tbl} AS a WHERE a.strCol IN (SELECT b.strCol FROM {tbl} AS b WHERE b.intCol = a.intCol + 1)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For subquery in WHERE with IN, can we also add NOT IN ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For IN clause subquery, let's also add test that the exact same query that uses ANY (supported by postgres) instead of IN returns identical results.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same goes for SOME I guess

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any and some will be tested in the function test instead of the table expression test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added. none are supported but added with ignore flag

},
{
"ignored": true,
"comments": "BETWEEN with a non-deterministic single-value result from sub-query is not supported",
"sql": "SELECT * FROM {tbl} AS a WHERE a.intCol BETWEEN (SELECT b.intCol FROM {tbl} AS b WHERE b.intCol = a.intCol + 1) AND 100"
},
{
"ignored": true,
"comments": "Relation Decorrelator not supported",
"sql": "SELECT * FROM {tbl} AS a WHERE a.intCol BETWEEN (SELECT MIN(b.intCol) FROM {tbl} AS b WHERE b.intCol = a.intCol + 1) AND 100"
},
{
"ignored": true,
"comments": "Relation Decorrelator not supported",
"sql": "SELECT * FROM {tbl} AS a WHERE EXISTS (SELECT strCol FROM {tbl} AS b WHERE b.intCol = a.intCol + 1)"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the same for NOT EXISTS ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added

},
{
"ignored": true,
"comments": "Relation Decorrelator not supported",
"sql": "SELECT * FROM {tbl} AS a WHERE NOT EXISTS (SELECT strCol FROM {tbl} AS b WHERE b.intCol = a.intCol + 1)"
},
{
"ignored": true,
"comments": "Relation Decorrelator not supported",
"sql": "SELECT * FROM {tbl} AS a WHERE (SELECT count(*) FROM {tbl} AS b WHERE b.intCol = a.intCol + 1) > 0"
}
]
},
"group_by_and_having_tests": {
"psql": "7.2.3",
"tables": {
"tbl1": {
"schema": [
{"name": "strCol", "type": "STRING"},
{"name": "intCol", "type": "INT"}
],
"inputs": [
["a", 3],
["b", 2],
["c", 5],
["a", 1]
]
},
"tbl2": {
"schema": [
{"name": "strCol1", "type": "STRING"},
{"name": "strCol2", "type": "STRING"},
{"name": "intCol", "type": "INT"}
],
"inputs": [
["a", "foo", 1],
["a", "bar", 2],
["b", "alice", 42],
["b", "bob", 196883]
]
}
},
"queries": [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also add the following ?

Non-correlated sub-query with HAVING

Without the subquery, we will first have to execute a query to find overall average and hardcode that value in HAVING clause of the second query.

SELECT group1, group2, AVG(col) AS groupAverage 
FROM FOO 
HAVING groupAverage > (
       SELECT AVG(col) FROM FOO)

Similarly correlated sub-query with HAVING

SELECT group1, group2, AVG(col) AS groupAverage 
FROM FOO as OUTER
HAVING groupAverage > (
                 SELECT AVG(col) 
                 FROM FOO 
                 WHERE group2 < OUTER.group2 )

I don't think correlated are supported yet though ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct correlate is not supported.

{ "sql": "SELECT strCol FROM {tbl1} GROUP BY strCol" },
{ "sql": "SELECT strCol FROM {tbl1} GROUP BY strCol, intCol" },
{ "sql": "SELECT strCol, intCol FROM {tbl1} GROUP BY strCol, intCol" },
{ "sql": "SELECT strCol, SUM(intCol) FROM {tbl1} GROUP BY strCol" },
{ "sql": "SELECT strCol, b.strCol2, (sum(a.intCol) * b.intCol) AS colAlias FROM {tbl1} a INNER JOIN {tbl2} b ON a.strCol = b.strCol1 GROUP BY strCol, b.strCol2, b.intCol" },
{ "sql": "SELECT MOD(b.intCol, 2), sum(a.intCol) AS colAlias FROM {tbl1} a INNER JOIN {tbl2} b ON a.strCol = b.strCol1 GROUP BY MOD(b.intCol, 2)" },
Copy link
Contributor

@siddharthteotia siddharthteotia Nov 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we also ensure that logically equivalent SQL queries return same results ?

For example INNER JOIN query and a corresponding subquery with IN clause should return same results.

SELECT p.product_id, p.product_name
FROM 
products p
INNER JOIN 
categories c 
ON p.category_id = c.category_id
WHERE c.category_id > 25

should return the same results as

SELECT p.product_id, p.product_name
FROM products p
WHERE p.category_id IN (
                    SELECT c.category_id
                    FROM categories c
                    WHERE c.category_id > 25
                 )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both are tested against a reference database. so yeah indirectly this is implied.

{ "sql": "SELECT strCol, SUM(intCol) FROM {tbl1} GROUP BY strCol HAVING SUM(intCol) > 3" },
{ "sql": "SELECT strCol, SUM(intCol) FROM {tbl1} GROUP BY strCol HAVING AVG(intCol) > 1 AND MIN(intCol) < 10" },
{ "sql": "SELECT strCol, b.strCol2, (sum(a.intCol) * b.intCol) AS colAlias FROM {tbl1} a INNER JOIN {tbl2} b ON a.strCol = b.strCol1 GROUP BY strCol, b.strCol2, b.intCol HAVING avg(a.intCol) < 100" },
{
"ignored": true,
"comment": "correlation is not supported. ALL is only supported if expand = false",
"sql": "SELECT strCol, SUM(intCol) FROM {tbl1} GROUP BY strCol HAVING AVG(intCol) >= ALL (SELECT AVG(intCol) FROM {tbl1} GROUP BY strCol)"
},
{
"ignored": true,
"comment": "correlation is not supported. SOME is only supported if expand = false",
"sql": "SELECT strCol, SUM(intCol) FROM {tbl1} GROUP BY strCol HAVING AVG(intCol) >= ANY (SELECT AVG(intCol) FROM {tbl1} GROUP BY strCol)"
}
]
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add select foo from (select 1) as foo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i am not sure this is a valid query in most other SQL systems. did you meant

select foo.* from (select 1) as foo

foo is a table not a column name in this context, and postgres returns a table type which is not supported in Pinot