Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: remove partition schema when incremental model overwrites partitions #620

Merged
merged 2 commits into from
Apr 9, 2024

Conversation

hiro-o918
Copy link
Contributor

Description

Glue has two kinds of schema, table schema and partition schema.
When the table schema is updated, the partition schema does not follow it automatically.
This causes HIVE_PARTITION_SCHEMA_MISMATCH in some conditions.

To avoid this, clean_up_partitions is modified to delete partition schemas as well as delete s3 objects.

I also found getting Parameters of table shows KeyError on some tables.
To fix it, I use get statement on table

Models used to test - Optional

Checklist

  • You followed contributing section
  • You kept your Pull Request small and focused on a single feature or bug fix.
  • You added unit testing when necessary
  • You added functional testing when necessary

@hiro-o918 hiro-o918 changed the title Remove partition schema when overwrite partition by incremental model feat: Remove partition schema when overwrite partition by incremental model Apr 9, 2024
@hiro-o918 hiro-o918 changed the title feat: Remove partition schema when overwrite partition by incremental model feat: remove partition schema when overwrite partition by incremental model Apr 9, 2024
@hiro-o918 hiro-o918 changed the title feat: remove partition schema when overwrite partition by incremental model feat: remove partition schema when incremental model overwrites partitions Apr 9, 2024
@nicor88
Copy link
Contributor

nicor88 commented Apr 9, 2024

Thanks for the fix @hiro-o918 💯

@nicor88 nicor88 merged commit 43b0bc2 into dbt-labs:main Apr 9, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants