Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingestion Pipeline pydantic validations w/ airflow apis #13823

Closed
pmbrull opened this issue Nov 2, 2023 · 0 comments · Fixed by #14461
Closed

Ingestion Pipeline pydantic validations w/ airflow apis #13823

pmbrull opened this issue Nov 2, 2023 · 0 comments · Fixed by #14461
Assignees
Labels
enhancement New feature or request Ingestion

Comments

@pmbrull
Copy link
Collaborator

pmbrull commented Nov 2, 2023

If we have misalignment in the ingestion pipeline JSON file that gets generated for the ingestion from OM UI and the pydantic models, the errors are hard to understand:

 IngestionPipeline expected: 170 validation errors for IngestionPipeline
sourceConfig -> config -> viewParsingTimeoutLimit
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> type
  value is not a valid enumeration member; permitted: 'DatabaseUsage' (type=type_error.enum; enum_values=[<DatabaseUsageConfigType.DatabaseUsage: 'DatabaseUsage'>])
sourceConfig -> config -> databaseFilterPattern
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> includeStoredProcedures
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> includeTables
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> includeTags
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> includeViews
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> markDeletedTables
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> queryParsingTimeoutLimit
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> schemaFilterPattern
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> tableFilterPattern
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> useFqnForFiltering
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> viewParsingTimeoutLimit
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> type
  value is not a valid enumeration member; permitted: 'DatabaseLineage' (type=type_error.enum; enum_values=[<DatabaseLineageConfigType.DatabaseLineage: 'DatabaseLineage'>])
sourceConfig -> config -> includeStoredProcedures
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> includeTables
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> includeTags
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> includeViews
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> markDeletedTables
  extra fields not permitted (type=value_error.extra)
sourceConfig -> config -> queryParsingTimeoutLimit
  extra fields not permitted (type=value_error.extra)

The goal of this ticket is to use the same parse_gracefully logic we have for workflows but at the Ingestion Pipeline level on the Airflow APIs.

@pmbrull pmbrull added enhancement New feature or request Ingestion labels Nov 2, 2023
@harshach harshach moved this to Ingestion - Bugs & Minor Features in Release 1.3.0 Nov 30, 2023
@TeddyCr TeddyCr moved this from Ingestion - Bugs & Minor Features to Done in Release 1.3.0 Dec 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Ingestion
Projects
No open projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants