Skip to content

Commit 28c3aa2

Browse files
cmcouto-silvacmcouto-silva
authored
Merge pull request #11 from glue-lab/feature/framework-enhancements
SQLDeps Framework Enhancement
2 parents 1c2d54f + 1b2f3b9 commit 28c3aa2

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+3030
-836
lines changed

‎.coveragerc

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[run]
2+
omit =
3+
sqldeps/app/*

‎.env.example

+7
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,13 @@ DB_NAME = "database"
55
DB_USER = "username"
66
DB_PASSWORD = "password"
77

8+
# Test database credentials
9+
TEST_DB_HOST = "host"
10+
TEST_DB_PORT = "5432"
11+
TEST_DB_NAME = "database"
12+
TEST_DB_USER = "username"
13+
TEST_DB_PASSWORD = "password"
14+
815
# API Keys
916
GROQ_API_KEY = "groq_token"
1017
OPENAI_API_KEY = "openai_token"

‎.github/workflows/ci.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ jobs:
1616
curl -LsSf https://astral.sh/uv/install.sh | sh
1717
echo "$HOME/.local/bin" >> $GITHUB_PATH
1818
19-
- name: Install dependencies
19+
- name: Install core dependencies
2020
run: uv sync
2121

2222
- name: Lint
@@ -62,8 +62,8 @@ jobs:
6262
- name: check python version
6363
run: uv run python --version
6464

65-
- name: Install dependencies
66-
run: uv sync
65+
- name: Install all dependencies (including optional)
66+
run: uv sync --all-extras
6767

6868
- name: Run tests
6969
run: uv run pytest

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,9 @@ cython_debug/
170170
# Ruff stuff:
171171
.ruff_cache/
172172

173+
# SQLDeps-specific cache:
174+
.sqldeps_cache
175+
173176
# PyPI configuration file
174177
.pypirc
175178

‎Makefile

+16
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,19 @@ fix:
77
check:
88
-uv run ruff format . --check
99
-uv run ruff check .
10+
11+
.PHONY: clean
12+
clean:
13+
# Remove Python cache directories
14+
find . -type d \( \
15+
-name "__pycache__" -o \
16+
-name "*.egg-info" -o \
17+
-name ".eggs" -o \
18+
-name ".ipynb_checkpoints" \
19+
\) -exec rm -rf {} +
20+
21+
# Remove compiled Python files
22+
find . -name "*.pyc" -delete
23+
24+
# Remove build, test, and cache directories
25+
rm -rf dist build htmlcov .pytest_cache .ruff_cache .sqldeps_cache .mypy_cache .tox 2>/dev/null || true

‎README.md

+55-1
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,55 @@ sqldeps data/sql_folder \
115115
-o folder_deps.csv
116116
```
117117

118+
```bash
119+
# Access CLI help
120+
sqldeps --help
121+
122+
Usage: sqldeps [OPTIONS] COMMAND [ARGS]...
123+
124+
SQL Dependency Extractor - Analyze SQL files to extract table and column dependencies
125+
126+
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
127+
│ --version Show the version and exit. │
128+
│ --install-completion Install completion for the current shell. │
129+
│ --show-completion Show completion for the current shell, to copy it or customize the installation. │
130+
│ --help Show this message and exit. │
131+
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
132+
╭─ Commands ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
133+
│ extract Extract SQL dependencies from file or folder. │
134+
│ app Run the SQLDeps web application │
135+
│ cache Manage SQLDeps cache │
136+
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
137+
138+
# Access CLI help for extraction
139+
$ sqldeps extract --help
140+
141+
Usage: sqldeps extract [OPTIONS] FPATH
142+
143+
Extract SQL dependencies from file or folder.
144+
This tool analyzes SQL files to identify table and column dependencies, optionally validating them against a real database schema.
145+
146+
╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
147+
│ * fpath PATH SQL file or directory path [default: None] [required] │
148+
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
149+
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
150+
│ --framework TEXT LLM framework to use [groq, openai, deepseek] [default: groq] │
151+
│ --model TEXT Model name for the selected framework [default: None] │
152+
│ --prompt FILE Path to custom prompt YAML file [default: None] │
153+
│ --recursive -r Recursively scan folder for SQL files │
154+
│ --db-match-schema --no-db-match-schema Match dependencies against database schema [default: no-db-match-schema] │
155+
│ --db-target-schemas TEXT Comma-separated list of target schemas to validate against [default: public] │
156+
│ --db-credentials FILE Path to database credentials YAML file [default: None] │
157+
│ --db-dialect TEXT Database dialect to use for schema validation [default: postgresql] │
158+
│ --n-workers INTEGER Number of workers for parallel processing. Use -1 for all CPU cores, 1 for sequential processing. [default: 1] │
159+
│ --rpm INTEGER Maximum requests per minute for API rate limiting (0 to disable) [default: 100] │
160+
│ --use-cache --no-use-cache Use local cache for SQL extraction results [default: use-cache] │
161+
│ --clear-cache --no-clear-cache Clear local cache after processing [default: no-clear-cache] │
162+
│ --output -o PATH Output file path for extracted dependencies [default: dependencies.json] │
163+
│ --help Show this message and exit. │
164+
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
165+
```
166+
118167
## Example
119168

120169
Given this SQL query:
@@ -249,7 +298,12 @@ The custom prompt YAML should include:
249298

250299
```yaml
251300
system_prompt: |
252-
Detailed instructions to the model...
301+
You are a SQL analyzer that extracts two key elements from SQL queries:
302+
303+
1. DEPENDENCIES: Tables and columns that must exist BEFORE query execution.
304+
2. OUTPUTS: Tables and columns permanently CREATED or MODIFIED by the query.
305+
306+
# Add detailed instructions for the LLM here...
253307
254308
user_prompt: |
255309
Extract SQL dependencies and outputs from this query:

‎pyproject.toml

+4-1
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ dependencies = [
3131
"pyyaml>=6.0.2",
3232
"sqlalchemy>=2.0.37",
3333
"sqlparse>=0.5.3",
34+
"tenacity>=9.0.0",
3435
"typer>=0.15.1",
3536
]
3637

@@ -48,6 +49,7 @@ dataviz = [
4849
[project.urls]
4950
Repository = "https://github.com/glue-lab/sqldeps"
5051
Documentation = "https://sqldeps.readthedocs.io"
52+
Questions = "https://github.com/glue-lab/sqldeps/discussions/categories/questions"
5153
Issues = "https://github.com/glue-lab/sqldeps/issues"
5254

5355
[project.scripts]
@@ -59,7 +61,7 @@ testpaths = ["tests"]
5961
markers = [
6062
"llm: marks tests that require LLM API calls (skipped by default)",
6163
]
62-
addopts = "-m 'not llm'"
64+
addopts = "-m 'not llm and not integration' --ignore=sqldeps/app"
6365

6466
[tool.hatch.metadata]
6567
allow-direct-references = false
@@ -69,6 +71,7 @@ packages = ["sqldeps"]
6971

7072
[dependency-groups]
7173
dev = [
74+
"pytest-cov>=6.0.0",
7275
"pytest>=8.3.4",
7376
"ruff>=0.9.7",
7477
]

‎scripts/.gitkeep

Whitespace-only changes.

‎scripts/evaluate.py

-120
This file was deleted.

‎scripts/extract_from_csv.py

-69
This file was deleted.

‎sqldeps/app/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""SQLDeps web application"""

‎sqldeps/app/assets/images/sqldeps_gray.png

43.8 KB
Loading

‎sqldeps/app/assets/images/sqldeps_white.png

42.5 KB
Loading

0 commit comments

Comments
 (0)