-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Teradata connector docs #16672
Merged
Merged
Add Teradata connector docs #16672
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
75 changes: 75 additions & 0 deletions
75
openmetadata-docs/content/v1.4.x/connectors/database/teradata/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,75 @@ | ||
--- | ||
title: Teradata | ||
slug: /connectors/database/teradata | ||
--- | ||
|
||
{% connectorDetailsHeader | ||
name="Teradata" | ||
stage="BETA" | ||
platform="OpenMetadata" | ||
availableFeatures=["Metadata", "Data Profiler"] | ||
unavailableFeatures=["Query Usage", "Data Quality", "Owners", "Tags", "Stored Procedures", "Lineage", "Column-level Lineage", "dbt"] | ||
/ %} | ||
|
||
In this section, we provide guides and references to use the Teradata connector. | ||
|
||
Configure and schedule Teradata metadata and profiler workflows from the OpenMetadata UI: | ||
|
||
- [Requirements](#requirements) | ||
- [Metadata Ingestion](#metadata-ingestion) | ||
- [Data Profiler](/connectors/ingestion/workflows/profiler) | ||
- [Data Quality](/connectors/ingestion/workflows/data-quality) | ||
|
||
{% partial file="/v1.4/connectors/ingestion-modes-tiles.md" variables={yamlPath: "/connectors/database/greenplum/yaml"} /%} | ||
|
||
## Requirements | ||
{%inlineCallout icon="description" bold="OpenMetadata 1.4 or later" href="/deployment"%} | ||
To deploy OpenMetadata, check the Deployment guides. | ||
{%/inlineCallout%} | ||
|
||
Connector was tested on Teradata DBS version 17.20. Since there are no significant changes in metadata objects, so it should work with 15.x, 16.x versions. | ||
|
||
|
||
## Metadata Ingestion | ||
|
||
By default, all valid users in Teradata DB has full access to metadata objects, so there are no any specific requirements to user privileges. | ||
|
||
{% partial | ||
file="/v1.4/connectors/metadata-ingestion-ui.md" | ||
variables={ | ||
connector: "Teradata", | ||
selectServicePath: "/images/v1.4/connectors/teradata/select-service.png", | ||
addNewServicePath: "/images/v1.4/connectors/teradata/add-new-service.png", | ||
serviceConnectionPath: "/images/v1.4/connectors/teradata/service-connection.png", | ||
} | ||
/%} | ||
|
||
{% stepsContainer %} | ||
{% extraContent parentTagName="stepsContainer" %} | ||
|
||
#### Connection Details | ||
|
||
- **Username**: Specify the User to connect to Teradata. | ||
- **Password**: Password to connect to Teradata | ||
- **Logmech**: Specifies the logon authentication method. Possible values are TD2 (the default), JWT, LDAP, KRB5 for Kerberos, or TDNEGO. | ||
- **LOGDATA**: Specifies additional data needed by a logon mechanism, such as a secure token, Distinguished Name, or a domain/realm name. LOGDATA values are specific to each logon mechanism. | ||
- **Host and Port**: Enter the fully qualified hostname and port number (default port for Teradata is 1025) for your Teradata deployment in the Host and Port field. | ||
- **Transaction Mode**: Specifies the transaction mode for the connection. Possible values are DEFAULT (the default), ANSI, or TERA. | ||
- **Teradata Database Account**: Specifies an account string to override the default account string defined for the database user. Accounts are used by the database for workload management and resource usage monitoring. | ||
- **Connection Options** and **Connection Arguments**: additional connection parameters. For more information please view teradatasql [docs](https://pypi.org/project/teradatasql/). | ||
|
||
{% partial file="/v1.4/connectors/database/advanced-configuration.md" /%} | ||
|
||
{% /extraContent %} | ||
|
||
{% partial file="/v1.4/connectors/test-connection.md" /%} | ||
|
||
{% partial file="/v1.4/connectors/database/configure-ingestion.md" /%} | ||
|
||
{% partial file="/v1.4/connectors/ingestion-schedule-and-deploy.md" /%} | ||
|
||
{% /stepsContainer %} | ||
|
||
{% partial file="/v1.4/connectors/troubleshooting.md" /%} | ||
|
||
{% partial file="/v1.4/connectors/database/related.md" /%} |
117 changes: 117 additions & 0 deletions
117
openmetadata-docs/content/v1.4.x/connectors/database/teradata/yaml.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
--- | ||
title: Run the Teradata Connector Externally | ||
slug: /connectors/database/teradata/yaml | ||
--- | ||
|
||
{% connectorDetailsHeader | ||
name="Teradata" | ||
stage="BETA" | ||
platform="OpenMetadata" | ||
availableFeatures=["Metadata", "Data Profiler", "Data Quality"] | ||
unavailableFeatures=["Query Usage", "Owners", "Tags", "Stored Procedures", "Lineage", "Column-level Lineage", "dbt"] | ||
/ %} | ||
|
||
In this section, we provide guides and references to use the Teradata connector. | ||
|
||
Configure and schedule Greenplum Teradata and profiler workflows from the OpenMetadata UI: | ||
|
||
- [Requirements](#requirements) | ||
- [Metadata Ingestion](#metadata-ingestion) | ||
- [Data Profiler](#data-profiler) | ||
- [Data Quality](#data-quality) | ||
|
||
|
||
{% partial file="/v1.4/connectors/external-ingestion-deployment.md" /%} | ||
|
||
## Requirements | ||
|
||
### Python Requirements | ||
|
||
{% partial file="/v1.4/connectors/python-requirements.md" /%} | ||
|
||
To run the Teradata ingestion, you will need to install: | ||
|
||
```bash | ||
pip3 install "openmetadata-ingestion[teradata]" | ||
``` | ||
|
||
## Metadata Ingestion | ||
|
||
All connectors are defined as JSON Schemas. | ||
[Here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/teradataConnection.json) | ||
you can find the structure to create a connection to Teradata. | ||
|
||
In order to create and run a Metadata Ingestion workflow, we will follow | ||
the steps to create a YAML configuration able to connect to the source, | ||
process the Entities if needed, and reach the OpenMetadata server. | ||
|
||
The workflow is modeled around the following | ||
[JSON Schema](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/workflow.json) | ||
|
||
### 1. Define the YAML Config | ||
|
||
This is a sample config for Teradata: | ||
|
||
{% codePreview %} | ||
|
||
{% codeInfoContainer %} | ||
|
||
#### Source Configuration - Service Connection | ||
|
||
{% codeInfo srNumber=1 %} | ||
|
||
**username**: Specify the User to connect to Teradata. | ||
|
||
{% /codeInfo %} | ||
{% codeInfo srNumber=2 %} | ||
|
||
**password**: User password to connect to Teradata | ||
|
||
{% /codeInfo %} | ||
|
||
{% codeInfo srNumber=3 %} | ||
|
||
**hostPort**: Enter the fully qualified hostname and port number for your Greenplum deployment in the Host and Port field. | ||
|
||
{% /codeInfo %} | ||
|
||
|
||
|
||
|
||
{% /codeInfoContainer %} | ||
|
||
{% codeBlock fileName="filename.yaml" %} | ||
|
||
```yaml {% isCodeBlock=true %} | ||
source: | ||
type: teradata | ||
serviceName: example_teradata | ||
serviceConnection: | ||
config: | ||
type: Teradata | ||
``` | ||
```yaml {% srNumber=1 %} | ||
username: username | ||
``` | ||
```yaml {% srNumber=2 %} | ||
password: <password> | ||
``` | ||
```yaml {% srNumber=3 %} | ||
hostPort: teradata:1025 | ||
``` | ||
|
||
{% partial file="/v1.4/connectors/yaml/database/source-config.md" /%} | ||
|
||
{% partial file="/v1.4/connectors/yaml/ingestion-sink.md" /%} | ||
|
||
{% partial file="/v1.4/connectors/yaml/workflow-config.md" /%} | ||
|
||
{% /codeBlock %} | ||
|
||
{% /codePreview %} | ||
|
||
{% partial file="/v1.4/connectors/yaml/ingestion-cli.md" /%} | ||
|
||
{% partial file="/v1.4/connectors/yaml/data-profiler.md" variables={connector: "teradata"} /%} | ||
|
||
{% partial file="/v1.4/connectors/yaml/data-quality.md" /%} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file added
BIN
+61.4 KB
openmetadata-docs/images/v1.4/connectors/teradata/add-new-service.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+326 KB
openmetadata-docs/images/v1.4/connectors/teradata/service-connection.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
73 changes: 73 additions & 0 deletions
73
openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database/Teradata.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,76 @@ | ||
# Teradata | ||
|
||
In this section, we provide guides and references to use the Teradata connector. | ||
|
||
## Requirements | ||
|
||
### Profiler & Data Quality | ||
|
||
Executing the profiler Workflow or data quality tests, will require the user to have `SELECT` permission on the tables/schemas where the profiler/tests will be executed. More information on the profiler workflow setup can be found [here](https://docs.open-metadata.org/connectors/ingestion/workflows/profiler) and data quality tests [here](https://docs.open-metadata.org/connectors/ingestion/workflows/data-quality). | ||
|
||
You can find further information on the Teradata connector in the [docs](https://docs.open-metadata.org/connectors/database/teradata). | ||
|
||
## Connection Details | ||
|
||
$$section | ||
### Connection Scheme $(id="scheme") | ||
|
||
SQLAlchemy driver scheme options. | ||
$$ | ||
|
||
$$section | ||
### Username $(id="username") | ||
|
||
Username to connect to Teradata. By default, any valid user has privileges to read all the metadata in Teradata. | ||
$$ | ||
|
||
$$section | ||
### Password $(id="password") | ||
|
||
Password to connect to Teradata. | ||
$$ | ||
|
||
$$section | ||
### Logon mechanism (Logmech) $(id="logmech") | ||
Specifies the logon authentication method. Possible values are TD2 (the default), JWT, LDAP, KRB5 for Kerberos, or TDNEGO. | ||
|
||
$$ | ||
|
||
$$section | ||
### Extra data for the chosen logon authentication method (LOGDATA) $(id="logdata") | ||
|
||
Specifies additional data needed by a logon mechanism, such as a secure token, Distinguished Name, or a domain/realm name. LOGDATA values are specific to each logon mechanism. | ||
$$ | ||
|
||
$$section | ||
### Host and Port $(id="hostPort") | ||
|
||
This parameter specifies the host and port of the Teradata instance. This should be specified as a string in the format `hostname:port`. For example, you might set the hostPort parameter to `example:1025`. | ||
$$ | ||
|
||
$$section | ||
### Transaction mode $(id="tmode") | ||
|
||
Specifies the transaction mode for the connection. Possible values are DEFAULT (the default), ANSI, or TERA. More information about this parameter can be found [here](https://teradata-docs.s3.amazonaws.com/doc/connectivity/jdbc/reference/current/jdbcug_chapter_2.html#TMODESEC) | ||
$$ | ||
|
||
$$section | ||
### Teradata Database account $(id="account") | ||
|
||
Specifies an account string to override the default account string defined for the database user. Accounts are used by the database for workload management and resource usage monitoring. | ||
|
||
$$ | ||
|
||
$$section | ||
### Connection Arguments $(id="connectionArguments") | ||
|
||
Additional connection arguments such as security or protocol configs that can be sent to service during connection. | ||
You can find further information about Connection Arguments [here](https://pypi.org/project/teradatasql/). | ||
$$ | ||
|
||
$$section | ||
### Connection Options $(id="connectionOptions") | ||
|
||
Additional connection options to build the URL that can be sent to service during the connection. | ||
You can find further information about Connection Options [here](https://pypi.org/project/teradatasql/). | ||
$$ |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gpby can we document what are the user permissions one needs to setup to get the metadata from Teradata?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By default all Teradata users has access to metadata objects (dbc.tablesv, dbc.columnsv, etc.) because public role has access to those objects. So DBA should do some explicit revokes to limit access to metadata object (e.g. KB0024146)