Fix parquet pruning when column names have periods #5710
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
alamb
commented
Mar 23, 2023
| @@ -384,7 +384,7 @@ fn build_statistics_record_batch<S: PruningStatistics>( | |||
| let mut arrays = Vec::<ArrayRef>::new(); | |||
| // For each needed statistics column: | |||
| for (column, statistics_type, stat_field) in required_columns.iter() { | |||
| let column = Column::from_qualified_name(column.name()); | |||
| let column = Column::from_name(column.name()); | |||
There was a problem hiding this comment.
This line is the bugfix
The rest of the PR is a test
thinkharderdev
approved these changes
Mar 24, 2023
viirya
reviewed
Mar 24, 2023
Comment on lines
+469
to
+470
| // note the column name has a period in it! | ||
| Field::new("service.name", service_name.data_type().clone(), true), |
viirya
reviewed
Mar 24, 2023
| Scenario::PeriodsInColumnNames, | ||
| "SELECT \"name\", \"service.name\" FROM t WHERE \"service.name\" = 'frontend' AND \"name\" != 'HTTP GET / DISPATCH'", | ||
| Some(0), | ||
| Some(2), // prune out middle and last row group |
There was a problem hiding this comment.
super nit:
Suggested change
| Some(2), // prune out middle and last row group | |
| Some(2), // prune out middle and last row group |
There was a problem hiding this comment.
sorry -- missed that -- will fix
viirya
approved these changes
Mar 24, 2023
crepererum
reviewed
Mar 27, 2023
For what it is worth, postgres uses all indexes (numbers) as I recall and that has its own set of horrible bugs (when the output indexes get mixed up). |
richox
pushed a commit
to richox/arrow-datafusion
that referenced
this pull request
Jun 16, 2023
…ache#5710)" This reverts commit 74c3955.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5708
Rationale for this change
We were seeing wrong results with parquet file that had columns with
.in their namesWhat changes are included in this PR?
Bug fix + test
Are these changes tested?
yes
I also verified that this fixes the issue we saw upstream in IOx https://github.com/influxdata/influxdb_iox/issues/7225#issuecomment-1481809300
Are there any user-facing changes?
bug fix
cc @crepererum