README.md

Overview

Versatile Data Kit (VDK) is a data framework that enables Data Engineers to

• 🧑‍💻develop,
• ▶️run,
• 📊and manage data workloads, aka data jobs

Its Lego-like design consists of lightweight Python modules installed via pip package manager. All VDK plugins are easy to combine.

VDK CLI can generate a data job and run your Python code and SQL queries.

🎯VDK SDK makes your code shorter, more readable, and faster to create.
🚦Ready-to-use data ETL/ELT patterns make Data Engineering with VDK efficient.

Data Engineers use VDK to implement automatic pull ingestion (E in ELT) and batch data transformation (T in ELT) into a database or any other data storage.

Data Journey and Versatile Data Kit

VDK creates data processing workflows to:

• Ingest data (extract)
• Transform data (transform)
• Export data (load)

Solve common data engineering problems

• Ingest data from different sources, including CSV files, JSON objects, and data from REST API services.
• Use Python/SQL and VDK templates to transform data.
• Ensure data applications are packaged, versioned, and deployed correctly while dealing with credentials, retries, and reconnects.
• Provide built-in monitoring and smart notification capabilities.
• Track both code and data modifications and the relationship between them, allowing quicker troubleshooting and version rollback.

What VDK can do

Getting Started

Installing VDK is a simple pip command. See the Getting Started guide to install VDK and create a data job.

Next Steps

• See use case examples that show how VDK fits into the data workflow.
• See the documentation for VDK.
• Read the article about using the Versatile Data Kit and Trino DB.
• Join us at a community meeting

Contributing

Create an issue or pull request on GitHub to submit suggestions or changes. If you are interested in contributing as a developer, visit the contributing page.

Contacts

• Connect on Slack by:
  • Joining the CNCF Slack workspace.
  • Joining the #versatile-data-kit channel.
• Follow us on Twitter.
• Subscribe to the Versatile Data Kit YouTube Channel.
• Join our development mailing list, used by developers and maintainers of VDK.

Code of Conduct

Everyone involved in working on the project's source code, or engaging in any issue trackers, Slack channels, and mailing lists is expected to be familiar with and follow the Code of Conduct.