vdk-confluence: add data source plugin for confluence (#3094)

Objective: Develop a plugin for data sourcing from Confluence,
incorporating an initial stream that comprehensively retrieves data
across all Confluence spaces.

Linked to: #3048
(additional tickets are added here)

Signed-off-by: Duygu Hasan [[email protected]](mailto:[email protected])

projects/vdk-plugins/vdk-confluence-data-source/.plugin-ci.yml

@@ -0,0 +1,22 @@
+# Copyright 2021-2024 VMware, Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+image: "python:3.7"
+
+.build-vdk-confluence-data-source:
+  variables:
+    PLUGIN_NAME: vdk-confluence-data-source
+  extends: .build-plugin
+
+build-py37-vdk-confluence-data-source:
+  extends: .build-vdk-confluence-data-source
+  image: "python:3.7"
+
+build-py311-vdk-confluence-data-source:
+  extends: .build-vdk-confluence-data-source
+  image: "python:3.11"
+
+release-vdk-confluence-data-source:
+  variables:
+    PLUGIN_NAME: vdk-confluence-data-source
+  extends: .release-plugin

projects/vdk-plugins/vdk-confluence-data-source/README.md

@@ -0,0 +1,93 @@
+# confluence-data-source
+
+**VDK data source plugin for Confluence**
+
+This plugin enables Versatile Data Kit (VDK) users to easily integrate and ingest data from Atlassian Confluence into their data processing jobs. It provides a streamlined way to access Confluence page content, utilizing the Confluence API to fetch pages from specified spaces or across the entire Confluence site. The plugin supports multiple authentication methods and is designed to work seamlessly with VDK, allowing for the configuration-driven ingestion of Confluence data.
+
+## Usage
+
+To use the `vdk-confluence-data-source` plugin, install it using pip:
+
+```
+pip install vdk-confluence-data-source
+```
+
+### Configuration
+
+Configuration options can be set in VDK by using the `vdk config-help` command, which provides a comprehensive list of all configurable options for your VDK installation. For the `confluence-data-source` plugin, the following configurations are available:
+
+### Configuration
+
+Configuration options for the `confluence-data-source` plugin can be set using VDK's configuration commands. These options allow you to specify the details required to connect to and interact with your Confluence instance. The available configurations are detailed below:
+
+| Name                    | Description                                                                                                                                   | Example Value                                                                                     |
+|-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
+| `confluence_url`        | The base URL of your Confluence instance.                                                                                                     | `"https://your-confluence-instance.atlassian.net/wiki"`                                           |
+| `space_key`             | The key of the Confluence space from which to fetch content. Leave empty to fetch from all spaces.                                            | `"DEMO"`                                                                                          |
+| `username`              | Confluence username for authentication. Required if using username and API token for authentication.                                          | `"[email protected]"`                                                                              |
+| `api_token`             | Confluence API token for authentication. Can be used instead of `personal_access_token`.                                                      | `"your_api_token"`                                                                                |
+| `personal_access_token` | Confluence personal access token for authentication. Can be used instead of `api_token`.                                                      | `"your_personal_access_token"`                                                                    |
+| `oauth2`                | OAuth2 credentials for authentication in dictionary format. Includes keys: 'access_token', 'access_token_secret', 'consumer_key', 'key_cert'. | `{"access_token": "...", "access_token_secret": "...", "consumer_key": "...", "key_cert": "..."}` |
+| `cloud`                 | Flag indicating if the Confluence instance is cloud-based.                                                                                    | `True`                                                                                            |
+| `confluence_kwargs`     | Additional keyword arguments for the Confluence client.                                                                                       | `{"timeout": 60}`                                                                                 |
+
+**Example usage in configuration:**
+
+```ini
+[confluence-data-source-config]
+confluence_url = https://your-confluence-instance.atlassian.net/wiki
+space_key = DEMO
+username = [email protected]
+api_token = your_api_token
+# personal_access_token = your_personal_access_token
+# oauth2 = {"access_token": "token", "access_token_secret": "secret", "consumer_key": "key", "key_cert": "cert"}
+cloud = True
+# confluence_kwargs = {"timeout": 60}
+```
+
+These configurations allow you to tailor the Confluence data source to your specific Confluence instance and authentication method, ensuring secure and efficient access to your Confluence content.
+### Example
+
+To use the `confluence-data-source` in a VDK job:
+
+```python
+from vdk.plugin.confluence_data_source import ConfluenceDataSource
+
+# Configuration example
+config = {
+    "confluence_url": "https://your-confluence-instance.atlassian.net/wiki",
+    "api_token": "your_api_token",
+    "space_key": "DEMO",
+}
+
+# Initialize the data source
+confluence_data_source = ConfluenceDataSource()
+confluence_data_source.configure(config)
+
+# Use the data source in your VDK job
+confluence_data_source.connect()
+for stream in confluence_data_source.streams():
+    for page_content in stream.read():
+        print(page_content.data, page_content.metadata)
+confluence_data_source.disconnect()
+```
+
+### Build and testing
+
+To build and test the `confluence-data-source` plugin, follow these steps:
+
+```
+pip install -r requirements.txt
+pip install -e .
+pytest
+```
+
+For plugins that are part of the [Versatile Data Kit Plugin repo](https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins), you can also use the `build-plugin.sh` script located in the VDK repo for building and testing:
+
+```
+../build-plugin.sh
+```
+
+#### Note about the CICD:
+
+For CI/CD, `.plugin-ci.yaml` is required for plugins that are part of the Versatile Data Kit Plugin repository. The CI/CD process is divided into two stages: build and release. Each stage consists of jobs that are configured to run on different Python versions (3.7, 3.8, 3.9, and 3.10), ensuring compatibility. The CI/CD runs according to rules designed to trigger the plugin's CI pipeline when changes are made within its directory.

projects/vdk-plugins/vdk-confluence-data-source/requirements.txt

@@ -0,0 +1,12 @@
+# this file is used to provide testing requirements
+# for requirements (dependencies) needed during and after installation of the plugin see (and update) setup.py install_requires section
+
+atlassian-python-api
+
+pytest
+pytest-mock
+python-dateutil
+requests-mock
+retrying
+vdk-core
+vdk-test-utils

projects/vdk-plugins/vdk-confluence-data-source/setup.py

@@ -0,0 +1,44 @@
+# Copyright 2021-2024 VMware, Inc.
+# SPDX-License-Identifier: Apache-2.0
+import pathlib
+
+import setuptools
+
+"""
+Builds a package with the help of setuptools in order for this package to be imported in other projects
+"""
+
+__version__ = "0.1.0"
+
+setuptools.setup(
+    name="vdk-confluence-data-source",
+    version=__version__,
+    url="https://github.com/vmware/versatile-data-kit",
+    description="VDK data source plugin for Confluence",
+    long_description=pathlib.Path("README.md").read_text(),
+    long_description_content_type="text/markdown",
+    install_requires=["vdk-core", "vdk-data-sources"],
+    package_dir={"": "src"},
+    packages=setuptools.find_namespace_packages(where="src"),
+    # This is the only vdk plugin specifc part
+    # Define entry point called "vdk.plugin.run" with name of plugin and module to act as entry point.
+    entry_points={
+        "vdk.plugin.run": [
+            "vdk-confluence-data-source = vdk.plugin.confluence_data_source.plugin_entry"
+        ]
+    },
+    classifiers=[
+        "Development Status :: 2 - Pre-Alpha",
+        "License :: OSI Approved :: Apache Software License",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
+    ],
+    project_urls={
+        "Documentation": "https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/vdk-confluence-data-source",
+        "Source Code": "https://github.com/vmware/versatile-data-kit/tree/main/projects/vdk-plugins/vdk-confluence-data-source",
+        "Bug Tracker": "https://github.com/vmware/versatile-data-kit/issues/new/choose",
+    },
+)