vdk-plugins: Move all plugins into separate directory

Until now, all plugins were stored inside the vdk-core directory.
This was counterintuitive, as plugins are separate packages from
vdk-core.
This change addresses this by moving all the plugins to a new
directory, and making the necessary changes to CI pipelines and
documentation.

Signed-off-by: Gabriel Georgiev <[email protected]>

.gitlab-ci.yml

@@ -8,6 +8,8 @@ include:
   - "projects/vdk-core/.gitlab-ci.yml"
   - "projects/vdk-heartbeat/.gitlab-ci.yml"
   - "projects/control-service/cicd/.gitlab-ci.yml"
+  - "projects/vdk-plugins/.plugin-common.yml"
+  - "projects/vdk-plugins/*/.plugin-ci.yml"
 
 stages:
   - build

CONTRIBUTING.md

@@ -11,7 +11,7 @@ For any questions about the CLA process, please refer to our [FAQ](https://cla.v
   * vdk-core - Python-based SDK containing a data library for developing and running data jobs. Includes a powerful plugin framework;
   * vdk-heartbeat - tool for verifying the deployed SDK and Control Service are functional and working correctly;
   * vdk-control-cli - User friendly CLI interface over Control Service operations including login/logout;
-  * vdk-core/plugins - Set of plugins that we maintain and provide for different use-cases like lineage, database support, etc.;
+  * vdk-plugins - Set of plugins that we maintain and provide for different use-cases like lineage, database support, etc.;
 * support - helper scripts used by developers of the project during their workday;
 * cicd - build and ci cd related scripts common across all projects. Each project also has its own cicd folder;
 * examples - list of example use-cases. Each example directory has its README with detailed explanations;

projects/vdk-core/.gitlab-ci.yml

@@ -1,9 +1,6 @@
 # Copyright 2021 VMware, Inc.
 # SPDX-License-Identifier: Apache-2.0
 
-include:
-  - "projects/vdk-core/plugins/.plugin-common.yml"
-  - "projects/vdk-core/plugins/*/.plugin-ci.yml"
 
 image: "python:3.9"
 

projects/vdk-core/CONTRIBUTING.md

@@ -20,7 +20,7 @@ Setting up the development environment can be done in the following way:
 
 ## Implementing plugins
 
-* You can find a template for implementing plugins in `/plugins/plugin-template`;
+* You can find a template for implementing plugins in `projects/vdk-plugins/plugin-template`;
 * Include your implementation files inside the `/src/vdk/internal/` directory, and any tests inside the `/tests/` directory;
 * Include your dependencies inside the `requirements.txt` file;
 * Change the name of the plugin package, the plugin itself and the name of the Python file containing the plugin hooks inside the `setup.py` file;

projects/vdk-core/README.md

@@ -30,7 +30,7 @@ vdk --help
 ## Plugins
 
 Plugins are a powerful way of extending or adapting Versatile Data Kit for all kinds of use-cases.
-For more information see [Plugins doc](./README_PLUGINS.md).
+For more information see [Plugins doc](../vdk-plugins/README_PLUGINS.md).
 
 ## Public interfaces
 

projects/vdk-core/cicd/build.sh

@@ -33,7 +33,7 @@ echo "Install the vdk-core in editable mode (develop mode)"
 pip install -e .
 
 echo "Install common vdk test utils library (in editable mode)"
-pip install -e plugins/vdk-test-utils
+pip install -e ../vdk-plugins/vdk-test-utils
 
 echo "Run unit tests and generate coverage report"
 pip install pytest-cov

projects/vdk-core/cicd/simple-functional-test.sh

@@ -54,19 +54,19 @@ python setup.py sdist --formats=gztar && pip install dist/* || fail "VDK Install
 
 function restore_vdk_in_editable_mode {
   pip install -e .
-  pip install --extra-index-url $PIP_EXTRA_INDEX_URL -e ./plugins/vdk-trino
-  pip install --extra-index-url $PIP_EXTRA_INDEX_URL -e ./plugins/vdk-plugin-control-cli
+  pip install --extra-index-url $PIP_EXTRA_INDEX_URL -e ./../vdk-plugins/vdk-trino
+  pip install --extra-index-url $PIP_EXTRA_INDEX_URL -e ./../vdk-plugins/vdk-plugin-control-cli
 }
 trap restore_vdk_in_editable_mode EXIT
 
 echo "Install database plugin for presto support"
-pushd plugins/vdk-trino || exit
+pushd ../vdk-plugins/vdk-trino || exit
 rm -rf dist/*
 python setup.py sdist --formats=gztar && pip install dist/* || fail "vdk-trino plugin Install failed"
 popd || exit
 
 echo "Install vdk-control-cli plugin"
-pushd plugins/vdk-plugin-control-cli || exit
+pushd ../vdk-plugins/vdk-plugin-control-cli || exit
 rm -rf dist/*
 python setup.py sdist --formats=gztar && pip install --extra-index-url $PIP_EXTRA_INDEX_URL dist/* || fail "vdk-control-cli plugin Install failed"
 popd || exit

projects/vdk-core/requirements.txt

@@ -15,7 +15,7 @@ pytest-sugar
 pytest-randomly
 smtpdfix
 pytest-cov
--e plugins/vdk-test-utils
+-e ../vdk-plugins/vdk-test-utils
 
 # Dependencies license report:
 pip-licenses

projects/vdk-core/plugins/.plugin-common.yml → projects/vdk-plugins/.plugin-common.yml

@@ -10,19 +10,19 @@
 .build-plugin:
   stage: build
   before_script:
-    - cd projects/vdk-core/
+    - cd projects/vdk-plugins/
     - pip install -U pip
-    - pip install ./plugins/vdk-test-utils # TODO: REMOVE THIS AT SOME POINT
+    - pip install ./vdk-test-utils # TODO: REMOVE THIS AT SOME POINT
   script:
     - echo "Build plugin $PLUGIN_NAME"
-    - cd plugins/$PLUGIN_NAME || exit 1
+    - cd ./$PLUGIN_NAME || exit 1
     - ../build-plugin.sh
   retry: !reference [.retry, retry_options]
   rules: # we want to trigger build jobs if there are changes to this plugin,
          # but not if there are changes to other plugins or the main directory
     - if: '$CI_COMMIT_BRANCH == "main" || $CI_PIPELINE_SOURCE == "external_pull_request_event"'
       changes:
-        - "projects/vdk-core/plugins/$PLUGIN_NAME/**/*"
+        - "projects/vdk-plugins/$PLUGIN_NAME/**/*"
   artifacts:
     when: always
     reports:
@@ -44,7 +44,7 @@
   rules:
     - if: '$CI_COMMIT_BRANCH == "main"'
       changes:
-       - "projects/vdk-core/plugins/$PLUGIN_NAME/**/*"
+       - "projects/vdk-plugins/$PLUGIN_NAME/**/*"
 
 
 .release-vdk-image:
@@ -73,4 +73,4 @@
   rules:
     - if: '$CI_COMMIT_BRANCH == "main"'
       changes:
-        - "projects/vdk-core/plugins/$PLUGIN_NAME/**/*"
+        - "projects/vdk-plugins/$PLUGIN_NAME/**/*"

projects/vdk-core/README_PLUGINS.md → projects/vdk-plugins/README.md

@@ -4,11 +4,11 @@ Plugins make Versatile Data Kit adaptable to any organization's use-cases.
 Example use-cases for plugins are different database connections, file systems, different transformation templates, or organization-specific best practices and patterns.
 
 
-![VDK Plugin Components](./docs/vdk-components.svg)
+![VDK Plugin Components](../vdk-core/docs/vdk-components.svg)
 
 ## List of plugins
 
-You can find a list of plugins that we have already developed in [plugins directory](./plugins).
+You can find a list of plugins that we have already developed in [plugins directory]().
 
 ## Installing and Using plugins
 
@@ -24,14 +24,14 @@ Once the plugin is installed, vdk automatically finds it and activates it.
 A plugin is a python module that enhances or changes the behavior of Versatile Data Kit. <br>
 A plugin is simply an implementation of one or more plugin hooks.
 
-See all supported hook function specifications in [specs.py](src/vdk/api/plugin/core_hook_spec.py).
+See all supported hook function specifications in [specs.py](../vdk-core/src/vdk/api/plugin/core_hook_spec.py).
 The spec documentation contains details and examples for how a hook can be used.
 
 To create a new plugin, there are only two steps necessary:<br>
 
 * Create your implementation of the plugin's hook(s):
   You will need to mark it with the `hookimpl` decorator.
-  Check out its [documentation here](src/vdk/api/plugin/hook_markers.py) to see how you can configure the hook execution order
+  Check out its [documentation here](../vdk-core/src/vdk/api/plugin/hook_markers.py) to see how you can configure the hook execution order
 ```python
 # this is module myproject.pluginmodule, which will be our plugin
 # define hookimpl as follows
@@ -77,21 +77,21 @@ As hook implementations can be functions (without class) it is recommended all h
 
 ### Generic Command-Line Lifecycle
 
-![plugin cli life cycle](docs/plugin-cli-lifecycle.svg)
+![plugin cli life cycle](../vdk-core/docs/plugin-cli-lifecycle.svg)
 
 Versatile Data Kit is used for executing different commands, some of them provided as plugins.
 Using the above hooks, one can extend the functionality of any command by adding monitoring, customizing logging, or adding new options.
 
-Check out the [CoreHookSpec class](src/vdk/api/plugin/core_hook_spec.py) documentation for more details.
+Check out the [CoreHookSpec class](../vdk-core/src/vdk/api/plugin/core_hook_spec.py) documentation for more details.
 
 ### Data Job Run (Execution) Cycle
 
-![plugin data job run cycle](docs/simple-data-job-lifecycle.svg)
+![plugin data job run cycle](../vdk-core/docs/simple-data-job-lifecycle.svg)
 
 The above image shows the normal run cycle of a data job. The hooks shown are only invoked when the "vdk run" command is invoked to execute a data job.
 
 
-Check out the [JobRunSpecs class](src/vdk/api/plugin/core_hook_spec.py) documentation for more details.
+Check out the [JobRunSpecs class](../vdk-core/src/vdk/api/plugin/core_hook_spec.py) documentation for more details.
 
 ## Public interfaces
 

projects/vdk-plugins/plugin-template/README.md

@@ -0,0 +1,20 @@
+This directory outlines a template which specifies the implementation of all vdk-core
+plugins. It includes a setup.py file, a /src/ directory containing all the plugin hooks
+and additional implementation files, a /tests/ directory containing all plugin-specific
+tests, and a .plugin-ci.yml file which specifies the CI/CD relevant to the plugins.
+
+The CI/CD is separated in two stages, a build stage and a release stage.
+The build stage is made up of three jobs, all which inherit from the same
+job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10).
+They run according to three rules, which are ordered in a way such that changes to a
+plugin's directory or the main directory triggers them, but changes to a different plugin
+do not.
+
+In order to add a new plugin , copy the plugin-template directory and follow the instructions in the files Generally those are
+
+* Update the setup.py file with correct name of the plugin;
+* Update `.plugin-ci.yml` file with name of the plugin - make sure to follow comments;
+* Include your implementation files inside the `src` folder;
+* Include any tests inside the `tests` so they can be ran by CI framework automatically.
+
+In order to build and test a plugin go to the plugin directory and use `../build-plugin.sh` script to build it