Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

documentation: Update to READMEs: Add a monthly download badge #2983

Merged
merged 5 commits into from
Jan 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
<img src="https://img.shields.io/github/commit-activity/m/vmware/versatile-data-kit" /></a>
<a href="https://github.com/vmware/versatile-data-kit/contributors" alt="Last Activity">
<img src="https://img.shields.io/github/last-commit/vmware/versatile-data-kit" alt="Last Activity"></a>
<a href="https://pypistats.org/packages/vdk-core" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-core.svg" alt="monthly download count for vdk-core">
<a href="https://github.com/vmware/versatile-data-kit/blob/main/LICENSE" alt="License">
<img src="https://img.shields.io/github/license/vmware/versatile-data-kit" alt="license"></a>
<a href="https://github.com/pre-commit/pre-commit">
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-control-cli/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Versatile Data Kit Control CLI

<a href="https://pypistats.org/packages/vdk-control-cli" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-control-cli.svg" alt="monthly download count for vdk-control-cli">

VDK Control CLI is meant for Data Engineers to use to manage the lifecycle of jobs - create, delete, deploy, configure Data Jobs.

To build or contribute, see [CONTRIBUTING.md](./CONTRIBUTING.md).
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-core/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Versatile Data Kit SDK Core

<a href="https://pypistats.org/packages/vdk-core" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-core.svg" alt="monthly download count for vdk-core">

To build or contribute, see [CONTRIBUTING.md](./CONTRIBUTING.md).

Versatile Data Kit enables data engineers, data scientists and data analysts to develop, deploy, run, and manage data processing workloads (called "Data Job").
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-heartbeat/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Versatile Data Kit Heartbeat tool

<a href="https://pypistats.org/packages/vdk-heartbeat" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-heartbeat.svg" alt="monthly download count for vdk-heartbeat">

Heartbeat tool for verifying deployed SDK and Control Service are functional and working correctly.<br>
It checks that a job can be created, deployed, run and deleted.

Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/airflow-provider-vdk/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Versatile Data Kit Airflow provider

<a href="https://pypistats.org/packages/airflow-provider-vdk" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/airflow-provider-vdk.svg" alt="monthly download count for airflow-provider-vdk">

A set of Airflow operators, sensors and a connection hook intended to help schedule Versatile Data Kit jobs using Apache Airflow.

# Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/quickstart-vdk/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Quickstart-VDK

<a href="https://pypistats.org/packages/quickstart-vdk" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/quickstart-vdk.svg" alt="monthly download count for quickstart-vdk">

This is the first VDK packaging that users would install to play around with.

It packages:
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-audit/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## Versatile Data Kit Audit Plugin

<a href="https://pypistats.org/packages/vdk-audit" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-audit.svg" alt="monthly download count for vdk-audit">

Visibility into the actions provides opportunities for test frameworks, logging
frameworks, and security tools to monitor and optionally limit actions taken by the
runtime.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-control-api-auth/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# vdk-control-api-auth

<a href="https://pypistats.org/packages/vdk-control-api-auth" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-control-api-auth.svg" alt="monthly download count for vdk-control-api-auth">

vdk-control-api-auth is a plugin library that implements authentication
utilities used by vdk-control-cli and other plugins.

Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-csv/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## Versatile Data Kit CSV Plugin

<a href="https://pypistats.org/packages/vdk-csv" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-csv.svg" alt="monthly download count for vdk-csv">

This plugin provides functionality to ingest and export CSV files.

### Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-dag/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# VDK DAGs

<a href="https://pypistats.org/packages/vdk-dag" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-dag.svg" alt="monthly download count for vdk-dag">

Express dependencies between data jobs.

A plugin for Versatile Data Kit extends its Job API with an additional feature that allows users to trigger so-called VDK DAGs.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-data-source-git/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# data-source-git

<a href="https://pypistats.org/packages/vdk-data-source-git" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-data-source-git.svg" alt="monthly download count for vdk-data-source-git">

Extracts content from Git repositories along with associated file metadata.

## Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-data-sources/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# data-sources

<a href="https://pypistats.org/packages/vdk-data-sources" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-data-sources.svg" alt="monthly download count for vdk-data-sources">

Enables Versatile Data Kit (VDK) to integrate with various data sources by providing a unified interface for data ingestion and management.

The data-sources project is a plugin for the Versatile Data Kit (VDK). It aims to simplify data ingestion from multiple sources by offering a single, unified API. Whether you're dealing with databases, REST APIs, or other forms of data, this project allows you to manage them all in a consistent manner. This is crucial for building scalable and maintainable data pipelines.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-duckdb/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# duckdb

<a href="https://pypistats.org/packages/vdk-duckdb" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-duckdb.svg" alt="monthly download count for vdk-duckdb">

DuckDB plugin for the Versatile Data Kit (VDK), which enables users to connect to and interact with DuckDB databases.
The purpose is to simplify data extraction, transformation, and loading tasks when working with DuckDB as a data source or destination

Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-gdp-execution-id/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-gdp-execution-id" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-gdp-execution-id.svg" alt="monthly download count for vdk-gdp-execution-id">

An installed Generative Data Pack plugin automatically expands the data sent for ingestion.

This GDP plugin detects the execution ID of a Data Job running, and decorates your data product with it. So that,
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-greenplum/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-greenplum" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-greenplum.svg" alt="monthly download count for vdk-greenplum">

This plugin allows vdk-core to interface with and execute queries against a Greenplum database.

# Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-huggingface/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Huggingface

<a href="https://pypistats.org/packages/vdk-huggingface" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-huggingface.svg" alt="monthly download count for vdk-huggingface">

Versatile Data Kit (VDK) plugin for integrating with Huggingface as both a data source and a target.
This plugin allows you to ingest data payloads into a Huggingface repository and makes it easier to work with datasets stored in Huggingface.

Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-impala/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
This plugin allows vdk-core to interface with and execute queries against an Impala database.

<a href="https://pypistats.org/packages/vdk-impala" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-impala.svg" alt="monthly download count for vdk-impala">

# Features

* It provides a powerful recovery mechanism handling a lot of challenges - like eventual consistency issues in Impala and more. In one production deployment of VDK it was able to improve SLA of Impala from 95% (queries directly to Impala) to 99% (queries using VDK to Impala).
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-ingest-file/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## VDK-INGEST-FILE Plugin

<a href="https://pypistats.org/packages/vdk-ingest-file" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-ingest-file.svg" alt="monthly download count for vdk-ingest-file">

This plugin provides functionality to ingest data into a file. It is intended for local testing.

### Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-ingest-http/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
### VDK-INGEST-HTTP Plugin

<a href="https://pypistats.org/packages/vdk-ingest-http" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-ingest-http.svg" alt="monthly download count for vdk-ingest-http">

This plugin provides functionality to ingest data over http.

### Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-ipython/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# vdk-ipython

<a href="https://pypistats.org/packages/vdk-ipython" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-ipython.svg" alt="monthly download count for vdk-ipython">

Ipython extension for VDK

This extension introduces a magic command for Jupyter.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-jobs-troubleshooting/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## VDK-JOBS-TROUBLESHOOTING Plugin

<a href="https://pypistats.org/packages/vdk-jobs-troubleshooting" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-jobs-troubleshooting.svg" alt="monthly download count for vdk-jobs-troubleshooting">

The VDK JOB Troubleshooting plugin provides the ability to add various troubleshooting utilities which can be accessed
during the data job runtime.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# vdk-jupyterlab-extension

<a href="https://pypistats.org/packages/vdk-jupyterlab-extension" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-jupyterlab-extension.svg" alt="monthly download count for vdk-jupyterlab-extension">

A Jupyterlab extension for using VDK
For more information see: https://github.com/vmware/versatile-data-kit/tree/main/specs/vep-994-jupyter-notebook-integration

Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-kerberos-auth/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-kerberos-auth" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-kerberos-auth.svg" alt="monthly download count for vdk-kerberos-auth">

The plugin provides GSSAPI Kerberos authentication on data job startup. The plugin also adds Kerberos/GSSAPI support for HTTP requests.

# Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-lineage-model/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# VDK Lineage Model

<a href="https://pypistats.org/packages/vdk-lineage-model" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-lineage-model.svg" alt="monthly download count for vdk-lineage-model">

VDK Lineage Model plugin aims to abstract emitting lineage data from VDK data jobs, so that different lineage loggers
can be configured at run time in any plugin that supports emitting lineage data.
The plugin describes the lineage data model and an interface for loggers that are responsible to send the lineage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-lineage/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# VDK Lineage

<a href="https://pypistats.org/packages/vdk-lineage" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-lineage.svg" alt="monthly download count for vdk-lineage">

VDK Lineage plugin provides lineage data (input data -> job -> output data) information and send it to a pre-configured
destination. The lineage data is send using [OpenLineage standard](https://openlineage.io)

Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-logging-format/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# This plugin has been deprecated; please use vdk-structlog instead.

<a href="https://pypistats.org/packages/vdk-logging-format" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-logging-format.svg" alt="monthly download count for vdk-logging-format">

This plugins allows for the configuration of the format of VDK logs.

# TEXT
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-meta-jobs/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Meta Jobs

<a href="https://pypistats.org/packages/vdk-meta-jobs" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-meta-jobs.svg" alt="monthly download count for vdk-meta-jobs">

Express dependencies between data jobs.

A plugin for Versatile Data Kit extends its Job API with an additional feature that allows users to trigger so-called Meta Jobs.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-notebook/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# vdk-notebook

<a href="https://pypistats.org/packages/vdk-notebook" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-notebook.svg" alt="monthly download count for vdk-notebook">

A new VDK plugin which supports running data jobs which consists of .ipynb files.
You can see [VDK Jupyter Integration VEP](https://github.com/vmware/versatile-data-kit/blob/main/specs/vep-994-jupyter-notebook-integration/README.md) for more information.

Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-oracle/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# oracle

<a href="https://pypistats.org/packages/vdk-oracle" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-oracle.svg" alt="monthly download count for vdk-oracle">

Support for VDK Managed Oracle connection

TODO: what the project is about, what is its purpose
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-plugin-control-cli/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-plugin-control-cli" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-plugin-control-cli.svg" alt="monthly download count for vdk-plugin-control-cli">

This plugin allows vdk-core to access vdk-control-cli functionality.

Upon installing it enhances vdk with CLI commands for data job lifecycle management available from vdk-control-cli package
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-postgres/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-postgres" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-postgres.svg" alt="monthly download count for vdk-postgres">

This plugin allows vdk-core to interface with and execute queries against a PostgreSQL database.

# Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-properties-fs/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-properties-fs" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-properties-fs.svg" alt="monthly download count for vdk-properties-fs">

This plugin allows vdk-core to read/write properties on the local FS. Mainly for development purposes,
to simplify a use-case with local Properties API usage, that would otherwise require a Control Service instance prerequisite.
For example, to quickly demo a data job that uses a secret, so that secret is quickly stored on the presenter's local FS.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-server/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## Versatile Data Kit Server Plugin

<a href="https://pypistats.org/packages/vdk-server" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-server.svg" alt="monthly download count for vdk-server">

This plugin facilitates installation of the Control Service in a local [Kind](https://kind.sigs.k8s.io/) cluster.

### Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-singer/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# singer

<a href="https://pypistats.org/packages/vdk-singer" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-singer.svg" alt="monthly download count for vdk-singer">

The vdk-singer plugin provides an easy way to integrate Singer Taps as data sources within the Versatile Data Kit (VDK).
This allows you to pull data from various external systems that have Singer Taps available and use them seamlessly
within your VDK pipelines.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-smarter/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# VDK Smarter

<a href="https://pypistats.org/packages/vdk-smarter" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-smarter.svg" alt="monthly download count for vdk-smarter">

Making VDK smarter by employing ML/AI.


Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-snowflake/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Versatile Data Kit Plugin for Snowflake Support

<a href="https://pypistats.org/packages/vdk-snowflake" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-snowflake.svg" alt="monthly download count for vdk-snowflake">

This plugin provides functionality, used by the Versatile Data Kit
to interact with a Snowflake instance. Users of the plugin can connect to
Snowflake, and execute queries against it.
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-sqlite/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-sqlite" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-sqlite.svg" alt="monthly download count for vdk-sqlite">

This plugin allows vdk-core to interface with, execute queries against, and ingest data to a SQLite database.

# Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-structlog/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Structured Logging For VDK

<a href="https://pypistats.org/packages/vdk-structlog" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-structlog.svg" alt="monthly download count for vdk-structlog">

This plugin allows users to:
- select the log output format
- configure the logging metadata
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-test-utils/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-test-utils" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-test-utils.svg" alt="monthly download count for vdk-test-utils">

This plugin provides utility tools used for the testing of vdk-core and vdk-core plugins.

# Usage
Expand Down
3 changes: 3 additions & 0 deletions projects/vdk-plugins/vdk-trino/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
<a href="https://pypistats.org/packages/vdk-trino" alt="Monthly Downloads">
<img src="https://img.shields.io/pypi/dm/vdk-trino.svg" alt="monthly download count for vdk-trino">

This plugin allows vdk-core to interface with and execute queries against a Trino database. Additionally, it can collect lineage data, assuming a lineage logger has been provided through the vdk-core configuration.


Expand Down
Loading