Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add sqlglot to parse sql dataflow #3310

Merged
merged 16 commits into from
Dec 31, 2024
Merged

feat: add sqlglot to parse sql dataflow #3310

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3cd733b
add sqlglot parsing
Light2Dark Dec 29, 2024
e3043f5
fix for comments and ddl's
Light2Dark Dec 29, 2024
2880dea
add support for dmls
Light2Dark Dec 30, 2024
1141567
modify comments and var name
Light2Dark Dec 30, 2024
f6f40f0
small refactor
Light2Dark Dec 30, 2024
7748a2a
adding sqlglot to dev dep
Light2Dark Dec 30, 2024
b1aff25
downgrade sqlglot min req
Light2Dark Dec 30, 2024
e091ba5
update sqlglot version to satisfiable
Light2Dark Dec 30, 2024
29b70ad
fix types and reduce sqlglot to min version
Light2Dark Dec 30, 2024
40f984e
remove unecc test
Light2Dark Dec 30, 2024
a38ac25
refactor flow
Light2Dark Dec 30, 2024
6c5b390
set duckdb to original
Light2Dark Dec 31, 2024
c4e7594
refactor test
Light2Dark Dec 31, 2024
a848267
add small test
Light2Dark Dec 31, 2024
40fdea9
remove test
Light2Dark Dec 31, 2024
436c0fb
add duckdb dialect and error handling for parse
Light2Dark Dec 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 0 additions & 12 deletions tests/_ast/test_sql_visitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -626,18 +626,6 @@ def test_find_sql_refs_nested_intersect() -> None:
"""
assert find_sql_refs(sql) == ["table2", "table3", "table4", "table1"]

@staticmethod
def test_find_sql_refs_with_recursive_cte() -> None:
sql = """
WITH RECURSIVE cte AS (
SELECT 1 AS n FROM table1
UNION ALL
SELECT n + 1 FROM cte WHERE n < 10
)
SELECT * FROM cte;
"""
assert find_sql_refs(sql) == ["table1"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is not yet supported, but you'd like to keep the test, you can decorate it with pytest.mark.xfail, as done here:

marimo/tests/_ast/test_sql_visitor.py

Line 569 in f8dc9f5

@pytest.mark.xfail(reason="Multiple CTEs are not supported")

Copy link
Collaborator Author

@Light2Dark Light2Dark Dec 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does work, I didn't see the need for it since we have tested nested cte's. Also noted on the dialect change. I may add back the has_duckdb dependency too.

thanks!!


@staticmethod
def test_find_sql_refs_with_alias() -> None:
sql = "SELECT * FROM employees AS e;"
Expand Down
Loading