Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HIVE-24167: Compilation fails due to equivalence mapping violation when CTE materialization is enabled #5452

Merged
merged 6 commits into from
Nov 15, 2024

Conversation

okumin
Copy link
Contributor

@okumin okumin commented Sep 17, 2024

What changes were proposed in this pull request?

With this change, Hive skips using or persisting runtime stats when PlanMapper throws equivalence mapping violation.

In some more contexts, we've tried to resolve the violation in #5037 and #5077. However, we couldn't find an obvious way to ideally avoid the issue, which means a modification will likely cause another problem. That's why we are currently biased toward the current approach.

https://issues.apache.org/jira/browse/HIVE-24167

Why are the changes needed?

When this issue occurs, the query always fails. We think the disadvantage of failure is not well balanced with the advantage of fast failure. A query might run with a bit poorer performance without runtime stats, but it is not critical in most cases. That's why we would try disabling runtime stats in this case.

Does this PR introduce any user-facing change?

In the new behavior, affected queries will not fail but run without runtime statistics.

Is the change a dependency upgrade?

No.

How was this patch tested?

qtests.

Copy link

@okumin okumin changed the title [WIP] HIVE-24167: TPC-DS query 14 fails while generating plan for the filter HIVE-24167: TPC-DS query 14 fails while generating plan for the filter Sep 18, 2024
@okumin okumin marked this pull request as ready for review September 18, 2024 12:27
Copy link
Contributor Author

@okumin okumin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing the PR. I am updating the branch based on the discussion.

@okumin
Copy link
Contributor Author

okumin commented Nov 5, 2024

@zabetak Updated. Thanks for reviewing this PR and giving many suggestions. I believe the change is much simpler

@okumin
Copy link
Contributor Author

okumin commented Nov 7, 2024

I rebased it again because the previous base revision was affected by HIVE-28611

Copy link

sonarqubecloud bot commented Nov 7, 2024

Copy link
Member

@zabetak zabetak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@okumin Now that you are a committer you can merge this yourself.

The PR/JIRA summary are a bit narrow given the nature of the problem and the fix that we are about to merge. Consider updating the commit message and JIRA with something more general/descriptive.

Examples
HIVE-24167: Compilation fails due to equivalence mapping violation when CTE materialization is enabled
HIVE-24167: Equivalence mapping violation in PlanMapper when CTE materialization is enabled

@okumin okumin changed the title HIVE-24167: TPC-DS query 14 fails while generating plan for the filter HIVE-24167: Compilation fails due to equivalence mapping violation when CTE materialization is enabled Nov 14, 2024
@okumin
Copy link
Contributor Author

okumin commented Nov 14, 2024

Thanks. Let me pick up the first option.

Ensure that the Pull Request has a +1 vote, and that 24 hours have elapsed since the first +1 vote was cast on the Pull Request. Note that this rule appears in the Hive Bylaws. Do not ignore it.

Based on this rule, I will wait for 1 day and return here.

@okumin okumin merged commit d85c239 into apache:master Nov 15, 2024
6 checks passed
@okumin okumin deleted the HIVE-24167-skip-backfill branch November 15, 2024 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants