Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[S3-Data-Lake] Make the Namespace/Database a required field #52639

Merged
merged 3 commits into from
Jan 31, 2025
+14 −13
Merged

[S3-Data-Lake] Make the Namespace/Database a required field #52639

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
19ff926
Make the Namespace/Database a required field
frifriSF59 Jan 29, 2025
7cdd36c
Update PR number
frifriSF59 Jan 29, 2025
243ec34
Expected spec and version 0.3.0
frifriSF59 Jan 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions ...c/main/kotlin/io/airbyte/cdk/load/command/iceberg/parquet/IcebergCatalogSpecifications.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ class NessieCatalogSpecification(
`Destination-defined` or `Source-defined`"""
)
@get:JsonProperty("namespace")
val namespace: String?
val namespace: String
) : CatalogType(catalogType)

/**
Expand Down Expand Up @@ -231,7 +231,7 @@ class GlueCatalogSpecification(
"""The Glue database name. This will ONLY be used if the `Destination Namespace` setting for the connection is set to `Destination-defined` or `Source-defined`"""
)
@get:JsonProperty("database_name")
val databaseName: String?
val databaseName: String
) : CatalogType(catalogType), AWSArnRoleSpecification

/**
Expand Down Expand Up @@ -266,7 +266,7 @@ class RestCatalogSpecification(
This will ONLY be used if the `Destination Namespace` setting for the connection is set to
`Destination-defined` or `Source-defined`"""
)
val namespace: String?
val namespace: String
) : CatalogType(catalogType)

/**
Expand Down Expand Up @@ -317,7 +317,7 @@ data class GlueCatalogConfiguration(
@get:JsonPropertyDescription(
"""The Glue database name. This will ONLY be used if the `Destination Namespace` setting for the connection is set to `Destination-defined` or `Source-defined`"""
)
val databaseName: String?
val databaseName: String
) : CatalogConfiguration, AWSArnRoleConfigurationProvider

/**
Expand All @@ -340,7 +340,7 @@ data class NessieCatalogConfiguration(
This will ONLY be used if the `Destination Namespace` setting for the connection is set to
`Destination-defined` or `Source-defined`"""
)
val namespace: String?
val namespace: String
) : CatalogConfiguration

/**
Expand All @@ -360,7 +360,7 @@ data class RestCatalogConfiguration(
This will ONLY be used if the `Destination Namespace` setting for the connection is set to
`Destination-defined` or `Source-defined`"""
)
val namespace: String?
val namespace: String
) : CatalogConfiguration

/**
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/destination-s3-data-lake/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ data:
alias: airbyte-connector-testing-secret-store
connectorType: destination
definitionId: 716ca874-520b-4902-9f80-9fad66754b89
dockerImageTag: 0.2.23
dockerImageTag: 0.3.0
dockerRepository: airbyte/destination-s3-data-lake
documentationUrl: https://docs.airbyte.com/integrations/destinations/s3-data-lake
githubIssueLabel: destination-s3-data-lake
Expand Down
6 changes: 3 additions & 3 deletions ...nnectors/destination-s3-data-lake/src/test-integration/resources/expected-spec-cloud.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@
"title" : "Namespace"
}
},
"required" : [ "catalog_type", "server_uri" ]
"required" : [ "catalog_type", "server_uri", "namespace" ]
}, {
"title" : "Glue Catalog",
"type" : "object",
Expand Down Expand Up @@ -122,7 +122,7 @@
"title" : "Database Name"
}
},
"required" : [ "catalog_type", "glue_id" ]
"required" : [ "catalog_type", "glue_id", "database_name" ]
}, {
"title" : "Rest Catalog",
"type" : "object",
Expand All @@ -146,7 +146,7 @@
"title" : "Namespace"
}
},
"required" : [ "catalog_type", "server_uri" ]
"required" : [ "catalog_type", "server_uri", "namespace" ]
} ],
"description" : "Specifies the type of Iceberg catalog (e.g., NESSIE, GLUE, REST) and its associated configuration.",
"title" : "Catalog Type",
Expand Down
6 changes: 3 additions & 3 deletions ...connectors/destination-s3-data-lake/src/test-integration/resources/expected-spec-oss.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@
"title" : "Namespace"
}
},
"required" : [ "catalog_type", "server_uri" ]
"required" : [ "catalog_type", "server_uri", "namespace" ]
}, {
"title" : "Glue Catalog",
"type" : "object",
Expand Down Expand Up @@ -122,7 +122,7 @@
"title" : "Database Name"
}
},
"required" : [ "catalog_type", "glue_id" ]
"required" : [ "catalog_type", "glue_id", "database_name" ]
}, {
"title" : "Rest Catalog",
"type" : "object",
Expand All @@ -146,7 +146,7 @@
"title" : "Namespace"
}
},
"required" : [ "catalog_type", "server_uri" ]
"required" : [ "catalog_type", "server_uri", "namespace" ]
} ],
"description" : "Specifies the type of Iceberg catalog (e.g., NESSIE, GLUE, REST) and its associated configuration.",
"title" : "Catalog Type",
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/destinations/s3-data-lake.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ for more information.

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------|
| 0.3.0 | 2025-01-31 | [\#52639](https://github.com/airbytehq/airbyte/pull/52639) | Make the database/namespace a required field |
| 0.2.23 | 2025-01-27 | [\#51600](https://github.com/airbytehq/airbyte/pull/51600) | Internal refactor |
| 0.2.22 | 2025-01-22 | [\#52081](https://github.com/airbytehq/airbyte/pull/52081) | Implement support for REST catalog |
| 0.2.21 | 2025-01-27 | [\#52564](https://github.com/airbytehq/airbyte/pull/52564) | Fix crash on stream with 0 records |
Expand Down
Loading