Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Update AthenaColumn to parse array types #672

Merged

Conversation

jeancochrane
Copy link
Contributor

Description

AthenaColumn is used to parse column metadata returned by Glue, particularly in AthenaAdapter.get_columns_in_relation(). The AthenaColumn.data_type property method is configured to convert a number of DDL data types to their DML equivalent (e.g. string ➡️ varchar), but it is not yet configured to convert array types, meaning that any code that consumes column definitions from get_columns_in_relation can raise errors like this one:

TYPE_MISMATCH: Unknown type: array<string>

This is causing problems for us as we begin adopting unit tests, since unit tests use get_columns_in_relation() in order to pull schema metadata for use in populating fixture data.

Models used to test - Optional

In addition to the unit tests I added as part of this PR, I manually tested this change against my team's dbt unit test suite to confirm that it resolves the TYPE_MISMATCH error. I'd be happy to pull together a minimal reproducible example if it would be useful.

Checklist

  • You followed contributing section
  • You kept your Pull Request small and focused on a single feature or bug fix.
  • You added unit testing when necessary
  • You added functional testing when necessary

dbt/adapters/athena/column.py Show resolved Hide resolved
dbt/adapters/athena/column.py Show resolved Hide resolved
dbt/adapters/athena/column.py Show resolved Hide resolved
dbt/adapters/athena/column.py Show resolved Hide resolved
tests/unit/test_column.py Show resolved Hide resolved
@jeancochrane jeancochrane changed the title Update AthenaColumn to parse array types feat: Update AthenaColumn to parse array types Jun 20, 2024
@nicor88
Copy link
Contributor

nicor88 commented Jun 24, 2024

Overall looks good to me, also the unit test suite seems quite complete.

@Jrmyy or @svdimchenko would you like to have a a look?

Copy link
Contributor

@nicor88 nicor88 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job @jeancochrane

@nicor88 nicor88 merged commit d95e9f1 into dbt-labs:main Jun 25, 2024
12 checks passed
kodiakhq bot referenced this pull request in cloudquery/policies Jul 1, 2024
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [dbt-athena-community](https://github.com/dbt-athena/dbt-athena) | patch | `==1.8.2` -> `==1.8.3` |

---

### Release Notes

<details>
<summary>dbt-athena/dbt-athena (dbt-athena-community)</summary>

### [`v1.8.3`](https://github.com/dbt-athena/dbt-athena/releases/tag/v1.8.3)

[Compare Source](https://github.com/dbt-athena/dbt-athena/compare/v1.8.2...v1.8.3)

#### What's Changed

##### Features

-   feat: Update `AthenaColumn` to parse array types by [@&#8203;jeancochrane](https://github.com/jeancochrane) in [https://github.com/dbt-athena/dbt-athena/pull/672](https://github.com/dbt-athena/dbt-athena/pull/672)

##### Fixes

-   fix: edge case fix on iceberg target table being dropped instead on being renamed first by [@&#8203;nicor88](https://github.com/nicor88) in [https://github.com/dbt-athena/dbt-athena/pull/667](https://github.com/dbt-athena/dbt-athena/pull/667)
-   fix: Improve caching of work_group lookup by [@&#8203;namelessjon](https://github.com/namelessjon) in [https://github.com/dbt-athena/dbt-athena/pull/678](https://github.com/dbt-athena/dbt-athena/pull/678)

##### Chore

-   chore: improve function tests for iceberg retries by [@&#8203;nicor88](https://github.com/nicor88) in [https://github.com/dbt-athena/dbt-athena/pull/674](https://github.com/dbt-athena/dbt-athena/pull/674)
-   chore: increase num_iceberg retries by [@&#8203;nicor88](https://github.com/nicor88) in [https://github.com/dbt-athena/dbt-athena/pull/679](https://github.com/dbt-athena/dbt-athena/pull/679)
-   chore: prepare release 1.8.3 by [@&#8203;nicor88](https://github.com/nicor88) in [https://github.com/dbt-athena/dbt-athena/pull/680](https://github.com/dbt-athena/dbt-athena/pull/680)

##### Dependencies

-   chore: Update moto requirement from ~=5.0.8 to ~=5.0.9 by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/dbt-athena/dbt-athena/pull/666](https://github.com/dbt-athena/dbt-athena/pull/666)
-   chore: Update pyupgrade requirement from ~=3.15 to ~=3.16 by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/dbt-athena/dbt-athena/pull/669](https://github.com/dbt-athena/dbt-athena/pull/669)
-   chore: Update flake8 requirement from ~=7.0 to ~=7.1 by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/dbt-athena/dbt-athena/pull/670](https://github.com/dbt-athena/dbt-athena/pull/670)
-   chore: Update dbt-tests-adapter requirement from ~=1.8.0 to ~=1.9.1 by [@&#8203;dependabot](https://github.com/dependabot) in [https://github.com/dbt-athena/dbt-athena/pull/673](https://github.com/dbt-athena/dbt-athena/pull/673)

##### Docs

-   chore: Added information about location table to readme ([#&#8203;628](https://github.com/dbt-athena/dbt-athena/issues/628)) by [@&#8203;cuff-links](https://github.com/cuff-links) in [https://github.com/dbt-athena/dbt-athena/pull/665](https://github.com/dbt-athena/dbt-athena/pull/665)

#### New Contributors

-   [@&#8203;cuff-links](https://github.com/cuff-links) made their first contribution in [https://github.com/dbt-athena/dbt-athena/pull/665](https://github.com/dbt-athena/dbt-athena/pull/665)
-   [@&#8203;namelessjon](https://github.com/namelessjon) made their first contribution in [https://github.com/dbt-athena/dbt-athena/pull/678](https://github.com/dbt-athena/dbt-athena/pull/678)

**Full Changelog**: dbt-labs/dbt-athena@v1.8.2...v1.8.3

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MjEuMiIsInVwZGF0ZWRJblZlciI6IjM3LjQyMS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWVyZ2UiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants