-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VDK-DuckDB: Introducing a new database plugin #2561
Conversation
d47e548
to
6e635fe
Compare
projects/vdk-plugins/vdk-duckdb/src/vdk/plugin/duckdb/ingest_to_duckdb.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-duckdb/src/vdk/plugin/duckdb/ingest_to_duckdb.py
Outdated
Show resolved
Hide resolved
projects/vdk-plugins/vdk-duckdb/src/vdk/plugin/duckdb/ingest_to_duckdb.py
Show resolved
Hide resolved
projects/vdk-plugins/vdk-duckdb/src/vdk/plugin/duckdb/ingest_to_duckdb.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall, the change looks ok. There are a few things you should do before merging:
- Rebase your branch to main. Right now you're submitting 22 commits. Ideally, it should be ~1.
git pull origin main --rebase
- Remove all the unnecessary files checked into version control, e.g.
.db
files - Change your commit message to use our commit template https://github.com/vmware/versatile-data-kit/blob/main/support/git-commit-template.txt
- Add a description to the PR (usually, it's fine to just copy the commit message if you're using the template)
- Address all comments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove also default_path.duckdb and temp.duckdb.
Normally, source code files are what you'd store in Git.
Auto-generated files, build artifacts or temporary files are not meant to be tracked in git. They are by-product of the code you write.
This enable us to keep the Git repository clean, organized, and efficient, adhering to best practices for source code management
742c3c1
to
4ce0d9a
Compare
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. Thanks for the contribution!
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
for more information, see https://pre-commit.ci
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
for more information, see https://pre-commit.ci
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
for more information, see https://pre-commit.ci
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
for more information, see https://pre-commit.ci
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
for more information, see https://pre-commit.ci
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
Created the basic plugin structure for DuckDB using the VDK CookieCutter with modifications.
723fd37
to
80f2d1c
Compare
Why?
This plugin has been created to check the differences how databases are implemented in a VDK plugin.
The aim of this change is to add DuckDB to the list of database plugins.
What?
I made a new DuckDB-plugin based on the SQLite-plugin.
For making the DuckDB work I had to change several things.
For the Plugin I had to change: Module Imports, Configuration, Connection Factory Method, Ingester Method and Library Specific Code.
For the Ingestion I had to change: Import Statements, Configuration and connection, Target Path, Data Type Conversion, Query compatibility,
General naming and references, Connection Handling.
For the Configuration I had to change: Constants, Class Names, Method for Database File, Configuration Key, Descriptions.
For the Connection I had to change: Classes, Class documentation, File Name, Import Statement, Connection Method, Isolation Level.
How has this been tested?
I've made a test program that first relied on the 'auto create feature' of the plugin. But after several errors I realized
that probably the auto create function wasn't functioning as expected.
So now the test doesn't rely on this function and instead doesn't create the table in advance, disables auto create, checks if the table is missing
and makes sure the ingestion logic remains.
What type of change are you making?
This is a new feature.
Now the users can make use of the DuckDB database from the list of database plugins.