Versatile Data Kit 0.13

Major features include:

New plugin: vdk-gdp-execution-id

An installed Generative Data Pack plugin automatically expands the data sent for ingestion.

This GDP plugin detects the execution ID of a Data Job running, and decorates your data product with it. So that,
it is now possible to correlate a data record with a particular ingestion Data Job execution ID.

For more information see the plugin documentation

vdk-dag: pass arguments to jobs in a DAG

Now each job in a DAG can be passed arguments :

{
"job_name": "name-of-job",
"team_name": "team-of-job",
"fail_meta_job_on_error": false,
"arguments": <ARGUMENTS IN DICTIONARY FORMAT HERE>,
"depends_on": ["name-of-job1", "name-of-job2"]
}

vdk-notebook: VDK job input in vdk cells

Users will be able to develop jobs entirely in a Notebook file with all features of VDK available out of the box
After installation of vdk-notebook users can now will have access to job_input interface to execute templates, ingest data and all else.

vdk-notebook: vdk and non-vdk cells

To enable separation of product and development code vdk-notebook integration provides a way for users to set which cells are deployable and part of their production code and which are not.

quickstart-vdk now includes the Operations UI

When installing quickstart-vdk VDK Server is available for local testing and now includes UI:

pip install quickstart-vdk
vdk server --install 

For more information see here

Versatile Data Kit Frontend npm libraries release

The Versatile Data Kit Frontend provides 2 npm (angular) libraries which can be used to build integrate VDK UI with your own screens:

• @versatiledatakit/data-pipelines
  Versatile Data Kit Data Pipelines library provides UI screens that helps to manage data jobs via Versatile Data Kit Control Service
• @versatiledatakit/shared
  Versatile Data Kit Shared library enables reusability of shared features like: NgRx Redux, Error Handlers, Utils, Generic Components, etc.

What's Changed

• control service: Add supported python version configuration by @doks5 in #1761
• control-service: fix python api release by @murphp15 in #1946
• control service: Dynamically set job base image in builder by @doks5 in #1864
• control-service: Add python_version to Control Service API by @doks5 in #1806
• control-service: Add python_version to Execution API by @mivanov1988 in #1878
• control-service: Add python_version to GraphQL API by @mivanov1988 in #1909
• control-service: Add support for Python 3.11 by @mivanov1988 in #1861
• control-service: Dynamically set vdk image in JobImageDeployer by @doks5 in #1883
• control-service: Expose supported python versions in helm by @doks5 in #1935
• control-service: Remove support for very old k8s apiVersion by @murphp15 in #1860
• control-service: add the frontend to helm by @murphp15 in #1885
• control-service: enable usage of aws temporary credentials by @mrMoZ1 in #1787
• control-service: expose supported python versions by @dakodakov in #1841
• control-service: fix failing image publisher by @murphp15 in #1810
• control-service: force job builder deploy by @mrMoZ1 in #1823
• control-service: new helm release by @murphp15 in #1910
• control-service: revert job builder python version by @mrMoZ1 in #1840
• control-service: update helm charts for service account credentials by @mrMoZ1 in #1800
• control-service: update job builders for aws temporary credentials by @mrMoZ1 in #1799
• documentation: VDK components explained by @ivakoleva in #1865
• frontend: Align code formatting in frontend projects by @gorankokin in #1863
• frontend: Configurable OAuth by @ivakoleva in #1913
• frontend: Update docs with build/test configuration by @DeltaMichael in #1928
• frontend: add build.sh by @tozka in #1807
• frontend: fix npm lint warnings by @DeltaMichael in #1808
• frontend: increase the amount of resources for build in cicd by @murphp15 in #1931
• frontend: prepare for official release shared and dp libs by @gorankokin in #1795
• frontend: publish docker image for ui by @DeltaMichael in #1872
• frontend: remove unused config in helm chart for frontend dns by @murphp15 in #1932
• frontend: Stabilization for e2e tests by @gorankokin in #1876
• frontend: Auth configurations organized by @ivakoleva in #1957
• frontend: change history link in data job by @gorankokin in #1884
• specs: VEP-1739 Update status and reorganise document by @doks5 in #1857
• specs: VEP-1739 updated API section by @mivanov1988 in #1882
• specs: update Multiple Python versions VEP summary by @tozka in #1792
• vdk-vep: update vep status by @dakodakov in #1951
• vdk-cicd: apply limit ranges for storage by @tozka in #1815
• vdk-cicd: set ephemeral storage request/limits by @tozka in #1813
• vdk-control-cli: fix circular import dependecy by @tozka in #1820
• vdk-control-cli: refactor output printing with printer class by @tozka in #1819
• vdk-control-cli: use assert_click_status by @tozka in #1817
• vdk-control-cli: use common output printer by @tozka in #1852
• vdk-control-cli: vdk list -mmm to return executions by @tozka in #1818
• vdk-control-service: publish python client library by @dakodakov in #1934
• vdk-dags: improve DAGs user-facing documentation by @yonitoo in #1892
• vdk-gdp-execution-id: a Generative Data Pack expanding with execution ID by @ivakoleva in #1877
• vdk-gdp-execution-id: import fix by @ivakoleva in #1961
• vdk-github-workflows: ubuntu latest update by @ivakoleva in #1943
• vdk-jupyter: UI test enhancements by @duyguHsnHsn in #1783
• vdk-jupyter: add UI vdk cell marks by @duyguHsnHsn in #1891
• vdk-jupyter: job run messages by @duyguHsnHsn in #1908
• vdk-jupyter: remove react-test-renderer package from package.json by @duyguHsnHsn in #1881
• vdk-lineage: support for latest version sqllineage library by @tozka in #1816
• vdk-meta-jobs: Meta Jobs DAG validation by @yonitoo in #1785
• vdk-meta-jobs: add DAG with args example by @yonitoo in #1859
• vdk-meta-jobs: add some configurable variable references in the VEP by @yonitoo in #1794
• vdk-meta-jobs: exec job with arguments by @yonitoo in #1839
• vdk-meta-jobs: fix DAG image in example by @yonitoo in #1920
• vdk-meta-jobs: improve DAGs code documentation by @yonitoo in #1873
• vdk-metajobs: Deprecate plugin by @gageorgiev in #1930
• vdk-notebook: add hook for saving error information into json file by @duyguHsnHsn in #1842
• vdk-plugins: add connection hook activity diagram by @tozka in #1786
• vdk-plugins: test only oldest and newest supported python version by @tozka in #1811
• vdk-server: quickstart vdk now includes the UI by @murphp15 in #1912

New Contributors

• @gorankokin made their first contribution in #1795

Full Changelog: v0.12...v0.13