-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Revert "Hive: close the fileIO client when closing the hive catalog" #11858
Conversation
…pache#10771)" This reverts commit 7e2920a.
c80bb0b
to
07ad68d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM for the revert. I will try to find out why he introduced this bug.
Thanks @Fokko!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Fokko! While the original fix intention was to close FileIO, seems like that's not happening correctly given the lifecycle/relation of HiveTableOperations to the HiveCatalog, and it's impacting operations! In the short term, agree with reverting, worst case is some uncleaned FileIOs but it won't impede jobs at least. Long term, will have to figure out how to track/close FileIO at the right time.
I'll go ahead and merge, thanks for reviewing @hussein-awala @bryanck ! |
Thanks for reviewing this @hussein-awala, @bryanck and @amogh-jahagirdar. I fully agree with Amoghs' assessment. Let me cherry-pick this to the |
This reverts commit 7e2920a from #10771.
Issue reported in #11783, #11633 and #11582.
I was able to reproduce this on PyIceberg apache/iceberg-python#1323, and while doing a
git bisect
, this led me to this PR:For now, I suggest reverting it to backport this to 1.7.2.
Closes #11783
Closes #11633
cc @hussein-awala @nastra @amogh-jahagirdar