Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Teradata connector docs #16672

Merged
merged 3 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
{% connectorInfoCard name="SingleStore" stage="PROD" href="/connectors/database/singlestore" platform="OpenMetadata" / %}
{% connectorInfoCard name="Snowflake" stage="PROD" href="/connectors/database/snowflake" platform="OpenMetadata" / %}
{% connectorInfoCard name="SQLite" stage="PROD" href="/connectors/database/sqlite" platform="OpenMetadata" / %}
{% connectorInfoCard name="Teradata" stage="PROD" href="/connectors/database/teradata" platform="OpenMetadata" / %}
{% connectorInfoCard name="Trino" stage="PROD" href="/connectors/database/trino" platform="OpenMetadata" / %}
{% connectorInfoCard name="Unity Catalog" stage="PROD" href="/connectors/database/unity-catalog" platform="OpenMetadata" / %}
{% connectorInfoCard name="Vertica" stage="PROD" href="/connectors/database/vertica" platform="OpenMetadata" / %}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
---
title: Teradata
slug: /connectors/database/teradata
---

{% connectorDetailsHeader
name="Teradata"
stage="BETA"
platform="OpenMetadata"
availableFeatures=["Metadata", "Data Profiler"]
unavailableFeatures=["Query Usage", "Data Quality", "Owners", "Tags", "Stored Procedures", "Lineage", "Column-level Lineage", "dbt"]
/ %}

In this section, we provide guides and references to use the Teradata connector.

Configure and schedule Teradata metadata and profiler workflows from the OpenMetadata UI:

- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Data Profiler](/connectors/ingestion/workflows/profiler)
- [Data Quality](/connectors/ingestion/workflows/data-quality)

{% partial file="/v1.4/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/greenplum/yaml"} /%}

## Requirements
{%inlineCallout icon="description" bold="OpenMetadata 1.4 or later" href="/deployment"%}
To deploy OpenMetadata, check the Deployment guides.
{%/inlineCallout%}

Connector was tested on Teradata DBS version 17.20. Since there are no significant changes in metadata objects, so it should work with 15.x, 16.x versions.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gpby can we document what are the user permissions one needs to setup to get the metadata from Teradata?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default all Teradata users has access to metadata objects (dbc.tablesv, dbc.columnsv, etc.) because public role has access to those objects. So DBA should do some explicit revokes to limit access to metadata object (e.g. KB0024146)



## Metadata Ingestion

By default, all valid users in Teradata DB has full access to metadata objects, so there are no any specific requirements to user privileges.

{% partial
file="/v1.4/connectors/metadata-ingestion-ui.md"
variables={
connector: "Teradata",
selectServicePath: "/images/v1.4/connectors/teradata/select-service.png",
addNewServicePath: "/images/v1.4/connectors/teradata/add-new-service.png",
serviceConnectionPath: "/images/v1.4/connectors/teradata/service-connection.png",
}
/%}

{% stepsContainer %}
{% extraContent parentTagName="stepsContainer" %}

#### Connection Details

- **Username**: Specify the User to connect to Teradata.
- **Password**: Password to connect to Teradata
- **Logmech**: Specifies the logon authentication method. Possible values are TD2 (the default), JWT, LDAP, KRB5 for Kerberos, or TDNEGO.
- **LOGDATA**: Specifies additional data needed by a logon mechanism, such as a secure token, Distinguished Name, or a domain/realm name. LOGDATA values are specific to each logon mechanism.
- **Host and Port**: Enter the fully qualified hostname and port number (default port for Teradata is 1025) for your Teradata deployment in the Host and Port field.
- **Transaction Mode**: Specifies the transaction mode for the connection. Possible values are DEFAULT (the default), ANSI, or TERA.
- **Teradata Database Account**: Specifies an account string to override the default account string defined for the database user. Accounts are used by the database for workload management and resource usage monitoring.
- **Connection Options** and **Connection Arguments**: additional connection parameters. For more information please view teradatasql [docs](https://pypi.org/project/teradatasql/).

{% partial file="/v1.4/connectors/database/advanced-configuration.md" /%}

{% /extraContent %}

{% partial file="/v1.4/connectors/test-connection.md" /%}

{% partial file="/v1.4/connectors/database/configure-ingestion.md" /%}

{% partial file="/v1.4/connectors/ingestion-schedule-and-deploy.md" /%}

{% /stepsContainer %}

{% partial file="/v1.4/connectors/troubleshooting.md" /%}

{% partial file="/v1.4/connectors/database/related.md" /%}
117 changes: 117 additions & 0 deletions openmetadata-docs/content/v1.4.x/connectors/database/teradata/yaml.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
---
title: Run the Teradata Connector Externally
slug: /connectors/database/teradata/yaml
---

{% connectorDetailsHeader
name="Teradata"
stage="BETA"
platform="OpenMetadata"
availableFeatures=["Metadata", "Data Profiler", "Data Quality"]
unavailableFeatures=["Query Usage", "Owners", "Tags", "Stored Procedures", "Lineage", "Column-level Lineage", "dbt"]
/ %}

In this section, we provide guides and references to use the Teradata connector.

Configure and schedule Greenplum Teradata and profiler workflows from the OpenMetadata UI:

- [Requirements](#requirements)
- [Metadata Ingestion](#metadata-ingestion)
- [Data Profiler](#data-profiler)
- [Data Quality](#data-quality)


{% partial file="/v1.4/connectors/external-ingestion-deployment.md" /%}

## Requirements

### Python Requirements

{% partial file="/v1.4/connectors/python-requirements.md" /%}

To run the Teradata ingestion, you will need to install:

```bash
pip3 install "openmetadata-ingestion[teradata]"
```

## Metadata Ingestion

All connectors are defined as JSON Schemas.
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/teradataConnection.json)
you can find the structure to create a connection to Teradata.

In order to create and run a Metadata Ingestion workflow, we will follow
the steps to create a YAML configuration able to connect to the source,
process the Entities if needed, and reach the OpenMetadata server.

The workflow is modeled around the following
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json)

### 1. Define the YAML Config

This is a sample config for Teradata:

{% codePreview %}

{% codeInfoContainer %}

#### Source Configuration - Service Connection

{% codeInfo srNumber=1 %}

**username**: Specify the User to connect to Teradata.

{% /codeInfo %}
{% codeInfo srNumber=2 %}

**password**: User password to connect to Teradata

{% /codeInfo %}

{% codeInfo srNumber=3 %}

**hostPort**: Enter the fully qualified hostname and port number for your Greenplum deployment in the Host and Port field.

{% /codeInfo %}




{% /codeInfoContainer %}

{% codeBlock fileName="filename.yaml" %}

```yaml {% isCodeBlock=true %}
source:
type: teradata
serviceName: example_teradata
serviceConnection:
config:
type: Teradata
```
```yaml {% srNumber=1 %}
username: username
```
```yaml {% srNumber=2 %}
password: <password>
```
```yaml {% srNumber=3 %}
hostPort: teradata:1025
```

{% partial file="/v1.4/connectors/yaml/database/source-config.md" /%}

{% partial file="/v1.4/connectors/yaml/ingestion-sink.md" /%}

{% partial file="/v1.4/connectors/yaml/workflow-config.md" /%}

{% /codeBlock %}

{% /codePreview %}

{% partial file="/v1.4/connectors/yaml/ingestion-cli.md" /%}

{% partial file="/v1.4/connectors/yaml/data-profiler.md" variables={connector: "teradata"} /%}

{% partial file="/v1.4/connectors/yaml/data-quality.md" /%}
4 changes: 4 additions & 0 deletions openmetadata-docs/content/v1.4.x/menu.md
Original file line number Diff line number Diff line change
Expand Up @@ -601,6 +601,10 @@ site_menu:
url: /connectors/database/sqlite
- category: Connectors / Database / SQLite / Run Externally
url: /connectors/database/sqlite/yaml
- category: Connectors / Database / Teradata
url: /connectors/database/teradata
- category: Connectors / Database / Teradata / Run Externally
url: /connectors/database/teradata/yaml
- category: Connectors / Database / Trino
url: /connectors/database/trino
- category: Connectors / Database / Trino / Run Externally
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -1,3 +1,76 @@
# Teradata

In this section, we provide guides and references to use the Teradata connector.

## Requirements

### Profiler & Data Quality

Executing the profiler Workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](https://docs.open-metadata.org/connectors/ingestion/workflows/profiler) and data quality tests [here](https://docs.open-metadata.org/connectors/ingestion/workflows/data-quality).

You can find further information on the Teradata connector in the [docs](https://docs.open-metadata.org/connectors/database/teradata).

## Connection Details

$$section
### Connection Scheme $(id="scheme")

SQLAlchemy driver scheme options.
$$

$$section
### Username $(id="username")

Username to connect to Teradata. By default, any valid user has privileges to read all the metadata in Teradata.
$$

$$section
### Password $(id="password")

Password to connect to Teradata.
$$

$$section
### Logon mechanism (Logmech) $(id="logmech")
Specifies the logon authentication method. Possible values are TD2 (the default), JWT, LDAP, KRB5 for Kerberos, or TDNEGO.

$$

$$section
### Extra data for the chosen logon authentication method (LOGDATA) $(id="logdata")

Specifies additional data needed by a logon mechanism, such as a secure token, Distinguished Name, or a domain/realm name. LOGDATA values are specific to each logon mechanism.
$$

$$section
### Host and Port $(id="hostPort")

This parameter specifies the host and port of the Teradata instance. This should be specified as a string in the format `hostname:port`. For example, you might set the hostPort parameter to `example:1025`.
$$

$$section
### Transaction mode $(id="tmode")

Specifies the transaction mode for the connection. Possible values are DEFAULT (the default), ANSI, or TERA. More information about this parameter can be found [here](https://teradata-docs.s3.amazonaws.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#TMODESEC)
$$

$$section
### Teradata Database account $(id="account")

Specifies an account string to override the default account string defined for the database user. Accounts are used by the database for workload management and resource usage monitoring.

$$

$$section
### Connection Arguments $(id="connectionArguments")

Additional connection arguments such as security or protocol configs that can be sent to service during connection.
You can find further information about Connection Arguments [here](https://pypi.org/project/teradatasql/).
$$

$$section
### Connection Options $(id="connectionOptions")

Additional connection options to build the URL that can be sent to service during the connection.
You can find further information about Connection Options [here](https://pypi.org/project/teradatasql/).
$$
Loading