Skip to content

Commit

Permalink
vdk-core: Remove reference to taurus from main namespace (#212)
Browse files Browse the repository at this point in the history
Until now, the main namespace of the vdk-core project was called
taurus. This is not ideal, as taurus is the name of an internal
VMware project, while Versatile Data Kit will soon be open source.

This change fixes that by renaming the taurus namespace to vdk,
and the sub-namespace which was previously called vdk to internal.

Testing done: pipelines passed

Signed-off-by: gageorgiev <[email protected]>

Co-authored-by: Antoni Ivanov <[email protected]>
  • Loading branch information
gabrielgeorgiev1 and antoniivanov authored Sep 10, 2021
1 parent 97d40f4 commit 977b8f9
Show file tree
Hide file tree
Showing 217 changed files with 960 additions and 852 deletions.
2 changes: 1 addition & 1 deletion projects/vdk-core/.gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

include:
- "projects/vdk-core/plugins/.plugin-common.yml"
- "projects/vdk-core/plugins/*/.plugin-ci.yml"
# - "projects/vdk-core/plugins/*/.plugin-ci.yml"

image: "python:3.9"

Expand Down
2 changes: 1 addition & 1 deletion projects/vdk-core/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Setting up the development environment can be done in the following way:
## Implementing plugins

* You can find a template for implementing plugins in `/plugins/plugin-template`;
* Include your implementation files inside the `/src/taurus/vdk/` directory, and any tests inside the `/tests/` directory;
* Include your implementation files inside the `/src/vdk/internal/` directory, and any tests inside the `/tests/` directory;
* Include your dependencies inside the `requirements.txt` file;
* Change the name of the plugin package, the plugin itself and the name of the Python file containing the plugin hooks inside the `setup.py` file;
* Change the name of the build and release jobs and the name variable inside the `.plugin-ci.yml` file to match the name of your plugin;
Expand Down
10 changes: 5 additions & 5 deletions projects/vdk-core/README_PLUGINS.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,20 +24,20 @@ Once the plugin is installed, vdk automatically finds it and activates it.
A plugin is a python module that enhances or changes the behavior of Versatile Data Kit. <br>
A plugin is simply an implementation of one or more plugin hooks.

See all supported hook function specifications in [specs.py](src/taurus/api/plugin/core_hook_spec.py).
See all supported hook function specifications in [specs.py](src/vdk/api/plugin/core_hook_spec.py).
The spec documentation contains details and examples for how a hook can be used.

To create a new plugin, there are only two steps necessary:<br>

* Create your implementation of the plugin's hook(s):
You will need to mark it with the `hookimpl` decorator.
Check out its [documentation here](src/taurus/api/plugin/hook_markers.py) to see how you can configure the hook execution order
Check out its [documentation here](src/vdk/api/plugin/hook_markers.py) to see how you can configure the hook execution order
```python
# this is module myproject.pluginmodule, which will be our plugin
# define hookimpl as follows

# you need to have vdk-core as dependency
from taurus.api.plugin.hook_markers import hookimpl
from vdk.api.plugin.hook_markers import hookimpl

# name of function must match name of hookspec function

Expand Down Expand Up @@ -78,12 +78,12 @@ entry_points={ 'vdk.plugin.run': ['name_of_plugin = myproject.pluginmodule'] }
Versatile Data Kit is used for executing different commands, some of them provided as plugins.
Using the above hooks, one can extend the functionality of any command by adding monitoring, customizing logging, or adding new options.

Check out the [CoreHookSpec class](src/taurus/api/plugin/core_hook_spec.py) documentation for more details.
Check out the [CoreHookSpec class](src/vdk/api/plugin/core_hook_spec.py) documentation for more details.

### Data Job Run (Execution) Cycle

![plugin data job run cycle](docs/simple-data-job-lifecycle.svg)

The above image shows the normal run cycle of a data job. The hooks shown are only invoked when the "vdk run" command is invoked to execute a data job.

Check out the [JobRunSpecs class](src/taurus/api/plugin/core_hook_spec.py) documentation for more details.
Check out the [JobRunSpecs class](src/vdk/api/plugin/core_hook_spec.py) documentation for more details.
2 changes: 1 addition & 1 deletion projects/vdk-core/cicd/release-vdk-core.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ cd ..

echo "$(cat version.txt | grep -o '[0-9]\.[0-9]').$CI_PIPELINE_ID" > version.txt

build_info_file="src/taurus/vdk/vdk_build_info.py"
build_info_file="src/vdk/internal/vdk_build_info.py"
echo "" > $build_info_file # clear build info file first

echo "RELEASE_VERSION='$(cat version.txt | grep -o '[0-9]\.[0-9]').$CI_PIPELINE_ID'" >> $build_info_file
Expand Down
16 changes: 8 additions & 8 deletions projects/vdk-core/cicd/simple-functional-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ python setup.py sdist --formats=gztar && pip install dist/* || fail "VDK Install

function restore_vdk_in_editable_mode {
pip install -e .
pip install -e --extra-index-url $PIP_EXTRA_INDEX_URL ./plugins/vdk-trino
pip install -e --extra-index-url $PIP_EXTRA_INDEX_URL ./plugins/vdk-plugin-control-cli
pip install --extra-index-url $PIP_EXTRA_INDEX_URL -e ./plugins/vdk-trino
# pip install --extra-index-url $PIP_EXTRA_INDEX_URL -e ./plugins/vdk-plugin-control-cli - disabled temporarily
}
trap restore_vdk_in_editable_mode EXIT

Expand All @@ -63,16 +63,16 @@ rm -rf dist/*
python setup.py sdist --formats=gztar && pip install dist/* || fail "vdk-trino plugin Install failed"
popd || exit

echo "Install vdk-control-cli plugin"
pushd plugins/vdk-plugin-control-cli || exit
rm -rf dist/*
python setup.py sdist --formats=gztar && pip install --extra-index-url $PIP_EXTRA_INDEX_URL dist/* || fail "vdk-control-cli plugin Install failed"
popd || exit
#echo "Install vdk-control-cli plugin"
#pushd plugins/vdk-plugin-control-cli || exit
#rm -rf dist/*
#python setup.py sdist --formats=gztar && pip install --extra-index-url $PIP_EXTRA_INDEX_URL dist/* || fail "vdk-control-cli plugin Install failed"
#popd || exit

echo "Run commands ..."
vdk || fail "vdk command failed"
vdk hello || fail "vdk hello command failed"
vdk deploy --help || fail "vdk-control-cli deploy --help command failed"
# vdk deploy --help || fail "vdk-control-cli deploy --help command failed"

vdk version || fail "vdk version failed"

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Copyright 2021 VMware, Inc.
# SPDX-License-Identifier: Apache-2.0
import click
from taurus.api.plugin.hook_markers import hookimpl
from taurus.api.plugin.plugin_utils import set_defaults_for_all_commands
from taurus.api.plugin.plugin_utils import set_defaults_for_specific_command
from vdk.api.plugin.hook_markers import hookimpl
from vdk.api.plugin.plugin_utils import set_defaults_for_all_commands
from vdk.api.plugin.plugin_utils import set_defaults_for_specific_command


@hookimpl
Expand Down
2 changes: 1 addition & 1 deletion projects/vdk-core/plugins/plugin-template/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
install_requires=["vdk-core"],
package_dir={"": "src"},
packages=setuptools.find_namespace_packages(where="src"),
entry_points={"vdk.plugin.run": ["plugin-template = taurus.vdk.plugin_template"]},
entry_points={"vdk.plugin.run": ["plugin-template = vdk.internal.plugin_template"]},
classifiers=[
"Development Status :: 4 - Beta",
"License :: OSI Approved :: Apache Software License",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Copyright 2021 VMware, Inc.
# SPDX-License-Identifier: Apache-2.0
from taurus.api.plugin.hook_markers import hookimpl
from vdk.api.plugin.hook_markers import hookimpl

"""
Include the plugin hooks here. For example:
from taurus.vdk.core.context import CoreContext
from vdk.internal.core.context import CoreContext
@hookimpl
def vdk_initialize(context: CoreContext):
Expand Down
2 changes: 1 addition & 1 deletion projects/vdk-core/plugins/vdk-csv/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
install_requires=["vdk-core", "pandas"],
package_dir={"": "src"},
packages=setuptools.find_namespace_packages(where="src"),
entry_points={"vdk.plugin.run": ["vdk-csv = taurus.vdk.csv_plugin"]},
entry_points={"vdk.plugin.run": ["vdk-csv = vdk.internal.csv_plugin"]},
classifiers=[
"Development Status :: 4 - Beta",
"License :: OSI Approved :: Apache Software License",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import pathlib
from typing import Dict

from taurus.api.job_input import IJobInput
from vdk.api.job_input import IJobInput

log = logging.getLogger(__name__)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@

import click
from click import Context
from taurus.api.plugin.hook_markers import hookimpl
from taurus.vdk.builtin_plugins.run.cli_run import run
from taurus.vdk.csv_ingest_job import csv_ingest_step
from vdk.api.plugin.hook_markers import hookimpl
from vdk.internal.builtin_plugins.run.cli_run import run
from vdk.internal.csv_ingest_job import csv_ingest_step

log = logging.getLogger(__name__)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
from unittest import mock

from click.testing import Result
from taurus.vdk import csv_plugin
from taurus.vdk.test_utils.util_funcs import cli_assert_equal
from taurus.vdk.test_utils.util_funcs import CliEntryBasedTestRunner
from taurus.vdk.test_utils.util_plugins import IngestIntoMemoryPlugin
from vdk.internal import csv_plugin
from vdk.internal.test_utils.util_funcs import cli_assert_equal
from vdk.internal.test_utils.util_funcs import CliEntryBasedTestRunner
from vdk.internal.test_utils.util_plugins import IngestIntoMemoryPlugin


def _get_file(filename):
Expand Down
2 changes: 1 addition & 1 deletion projects/vdk-core/plugins/vdk-ingest-file/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package_dir={"": "src"},
packages=setuptools.find_namespace_packages(where="src"),
entry_points={
"vdk.plugin.run": ["vdk-ingest-file = taurus.vdk.ingest_file_plugin"]
"vdk.plugin.run": ["vdk-ingest-file = vdk.internal.ingest_file_plugin"]
},
classifiers=[
"Development Status :: 4 - Beta",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
"""
import logging

from taurus.api.plugin.hook_markers import hookimpl
from taurus.vdk.builtin_plugins.ingestion.ingester_base import IIngesterPlugin
from taurus.vdk.builtin_plugins.run.job_context import JobContext
from taurus.vdk.core.config import ConfigurationBuilder
from taurus.vdk.ingestion_to_file import IngestionToFile
from vdk.api.plugin.hook_markers import hookimpl
from vdk.internal.builtin_plugins.ingestion.ingester_base import IIngesterPlugin
from vdk.internal.builtin_plugins.run.job_context import JobContext
from vdk.internal.core.config import ConfigurationBuilder
from vdk.internal.ingestion_to_file import IngestionToFile


log = logging.getLogger(__name__)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from typing import List
from typing import Optional

from taurus.api.plugin.plugin_input import IIngesterPlugin
from vdk.api.plugin.plugin_input import IIngesterPlugin


log = logging.getLogger(__name__)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
from unittest.mock import mock_open
from unittest.mock import patch

from taurus.vdk.ingestion_to_file import IngestionToFile
from vdk.internal.ingestion_to_file import IngestionToFile


def test_ingestion_to_file():
Expand Down
2 changes: 1 addition & 1 deletion projects/vdk-core/plugins/vdk-ingest-http/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
package_dir={"": "src"},
packages=setuptools.find_namespace_packages(where="src"),
entry_points={
"vdk.plugin.run": ["vdk-ingest-http = taurus.vdk.ingest_http_plugin"]
"vdk.plugin.run": ["vdk-ingest-http = vdk.internal.ingest_http_plugin"]
},
classifiers=[
"Development Status :: 4 - Beta",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
"""
import logging

from taurus.api.plugin.hook_markers import hookimpl
from taurus.vdk.builtin_plugins.ingestion.ingester_base import IIngesterPlugin
from taurus.vdk.builtin_plugins.run.job_context import JobContext
from taurus.vdk.ingest_over_http import IngestOverHttp
from vdk.api.plugin.hook_markers import hookimpl
from vdk.internal.builtin_plugins.ingestion.ingester_base import IIngesterPlugin
from vdk.internal.builtin_plugins.run.job_context import JobContext
from vdk.internal.ingest_over_http import IngestOverHttp


log = logging.getLogger(__name__)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
from typing import Optional

import requests
from taurus.vdk.builtin_plugins.ingestion.ingester_base import IIngesterPlugin
from taurus.vdk.core import errors
from vdk.internal.builtin_plugins.ingestion.ingester_base import IIngesterPlugin
from vdk.internal.core import errors

log = logging.getLogger(__name__)

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Copyright 2021 VMware, Inc.
# SPDX-License-Identifier: Apache-2.0
from taurus.api.job_input import IJobInput
from vdk.api.job_input import IJobInput


def run(job_input: IJobInput):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@

from click.testing import Result
from pytest_httpserver.pytest_plugin import PluginHTTPServer
from taurus.vdk import ingest_http_plugin
from taurus.vdk.test_utils.util_funcs import cli_assert_equal
from taurus.vdk.test_utils.util_funcs import CliEntryBasedTestRunner
from taurus.vdk.test_utils.util_funcs import get_test_job_path
from vdk.internal import ingest_http_plugin
from vdk.internal.test_utils.util_funcs import cli_assert_equal
from vdk.internal.test_utils.util_funcs import CliEntryBasedTestRunner
from vdk.internal.test_utils.util_funcs import get_test_job_path
from werkzeug import Response


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
from unittest import mock

import pytest
from taurus.api.plugin.plugin_registry import PluginException
from taurus.vdk.core.errors import PlatformServiceError
from taurus.vdk.core.errors import UserCodeError
from taurus.vdk.core.errors import VdkConfigurationError
from taurus.vdk.ingest_over_http import IngestOverHttp
from vdk.api.plugin.plugin_registry import PluginException
from vdk.internal.core.errors import PlatformServiceError
from vdk.internal.core.errors import UserCodeError
from vdk.internal.core.errors import VdkConfigurationError
from vdk.internal.ingest_over_http import IngestOverHttp

payload: dict = {
"@id": "test_id",
Expand Down
4 changes: 2 additions & 2 deletions projects/vdk-core/plugins/vdk-logging-json/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,6 @@ pip install vdk-logging-json

And all logs will be automatically formatted to JSON. They will appear like this:
```
{"@timestamp":"2021-08-04T12:51:11.532Z","created":"1628070671","jobname":"example-job","level":"DEBUG","modulename":"taurus.vdk.trino_connection","filename":"managed_connection_base.py","lineno":"69","funcname":"connect","attemptid":"1628070671-452613-739749","message":"Established <trino.dbapi.Connection object at 0x10b9b1d30>"}
{"@timestamp":"2021-08-04T12:51:11.532Z","created:1628070671","jobname":"example-job","level":"DEBUG","modulename":"taurus.vdk.trino_connection","filename":"managed_cursor.py","lineno":"29","funcname":"execute","attemptid":"1628070671-452613-739749","message":"Executing query: select 1"}
{"@timestamp":"2021-08-04T12:51:11.532Z","created":"1628070671","jobname":"example-job","level":"DEBUG","modulename":"vdk.internal.trino_connection","filename":"managed_connection_base.py","lineno":"69","funcname":"connect","attemptid":"1628070671-452613-739749","message":"Established <trino.dbapi.Connection object at 0x10b9b1d30>"}
{"@timestamp":"2021-08-04T12:51:11.532Z","created:1628070671","jobname":"example-job","level":"DEBUG","modulename":"vdk.internal.trino_connection","filename":"managed_cursor.py","lineno":"29","funcname":"execute","attemptid":"1628070671-452613-739749","message":"Executing query: select 1"}
```
2 changes: 1 addition & 1 deletion projects/vdk-core/plugins/vdk-logging-json/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
install_requires=["vdk-core"],
package_dir={"": "src"},
packages=setuptools.find_namespace_packages(where="src"),
entry_points={"vdk.plugin.run": ["vdk-logging-json = taurus.vdk.logging_json"]},
entry_points={"vdk.plugin.run": ["vdk-logging-json = vdk.internal.logging_json"]},
classifiers=[
"Development Status :: 4 - Beta",
"License :: OSI Approved :: Apache Software License",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
# SPDX-License-Identifier: Apache-2.0
import logging

from taurus.api.plugin.hook_markers import hookimpl
from taurus.vdk.builtin_plugins.run.job_context import JobContext
from taurus.vdk.core.statestore import CommonStoreKeys
from vdk.api.plugin.hook_markers import hookimpl
from vdk.internal.builtin_plugins.run.job_context import JobContext
from vdk.internal.core.statestore import CommonStoreKeys

# the labels follow the labelling recommendations found here: http://ltsv.org/
format_template = (
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# SPDX-License-Identifier: Apache-2.0
import logging

from taurus.vdk.logging_json import format_template
from vdk.internal.logging_json import format_template


def test_json_logging():
Expand Down
4 changes: 2 additions & 2 deletions projects/vdk-core/plugins/vdk-logging-ltsv/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,6 @@ pip install vdk-logging-ltsv

And all logs will be automatically formatted to LTSV format. They would start to look something like this:
```
@timestamp:2021-08-04T12:51:11 created:1628070671 jobname:example-job level:DEBUG modulename:taurus.vdk.trino_connection filename:managed_connection_base.py lineno:69 funcname:connect attemptid:1628070671-452613-739749 message:Established <trino.dbapi.Connection object at 0x10b9b1d30>
@timestamp:2021-08-04T12:51:11 created:1628070671 jobname:example-job level:DEBUG modulename:taurus.vdk.trino_connection filename:managed_cursor.py lineno:29 funcname:execute attemptid:1628070671-452613-739749 message:Executing query: select 1
@timestamp:2021-08-04T12:51:11 created:1628070671 jobname:example-job level:DEBUG modulename:vdk.internal.trino_connection filename:managed_connection_base.py lineno:69 funcname:connect attemptid:1628070671-452613-739749 message:Established <trino.dbapi.Connection object at 0x10b9b1d30>
@timestamp:2021-08-04T12:51:11 created:1628070671 jobname:example-job level:DEBUG modulename:vdk.internal.trino_connection filename:managed_cursor.py lineno:29 funcname:execute attemptid:1628070671-452613-739749 message:Executing query: select 1
```
2 changes: 1 addition & 1 deletion projects/vdk-core/plugins/vdk-logging-ltsv/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
install_requires=["vdk-core"],
package_dir={"": "src"},
packages=setuptools.find_namespace_packages(where="src"),
entry_points={"vdk.plugin.run": ["vdk-logging-ltsv = taurus.vdk.logging_ltsv"]},
entry_points={"vdk.plugin.run": ["vdk-logging-ltsv = vdk.internal.logging_ltsv"]},
classifiers=[
"Development Status :: 4 - Beta",
"License :: OSI Approved :: Apache Software License",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
# SPDX-License-Identifier: Apache-2.0
import logging

from taurus.api.plugin.hook_markers import hookimpl
from taurus.vdk.builtin_plugins.run.job_context import JobContext
from taurus.vdk.core.statestore import CommonStoreKeys
from vdk.api.plugin.hook_markers import hookimpl
from vdk.internal.builtin_plugins.run.job_context import JobContext
from vdk.internal.core.statestore import CommonStoreKeys

# the labels follow the labelling recommendations found here: http://ltsv.org/
format_template = (
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# SPDX-License-Identifier: Apache-2.0
import logging

from taurus.vdk.logging_ltsv import format_template
from vdk.internal.logging_ltsv import format_template


def test_ltsv_logging():
Expand Down
4 changes: 2 additions & 2 deletions projects/vdk-core/plugins/vdk-plugin-control-cli/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@
packages=setuptools.find_namespace_packages(where="src"),
entry_points={
"vdk.plugin.run": [
"vdk-plugin-control-cli = taurus.vdk.vdk_plugin_control_cli",
"vdk-control-service-properties = taurus.vdk.properties_plugin",
"vdk-plugin-control-cli = vdk.internal.vdk_plugin_control_cli",
"vdk-control-service-properties = vdk.internal.properties_plugin",
]
},
classifiers=[
Expand Down
Loading

0 comments on commit 977b8f9

Please sign in to comment.