Refactor timeValueCheck for Pinot TableConfig #9349
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jackie-Jiang
reviewed
Sep 9, 2022
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/IndexingConfig.java
Outdated
Show resolved
Hide resolved
xiangfu0
force-pushed
the
skip_time_value_check
branch
4 times, most recently
from
19371ef to
02cb9d8
Compare
xiangfu0
changed the title
xiangfu0
force-pushed
the
skip_time_value_check
branch
2 times, most recently
from
116db56 to
1fbe2b8
Compare
|
The only issue I see is these two configs will be pretty difficult to understand for a new user. e.g. why are there two time checks in record and segment? What are records in the first place? |
xiangfu0
force-pushed
the
skip_time_value_check
branch
6 times, most recently
from
8f54296 to
074becf
Compare
xiangfu0
force-pushed
the
skip_time_value_check
branch
from
074becf to
3ed8662
Compare
Codecov Report
@@ Coverage Diff @@
## master #9349 +/- ##
=============================================
- Coverage 63.38% 35.02% -28.37%
+ Complexity 4755 185 -4570
=============================================
Files 1831 1884 +53
Lines 97999 100273 +2274
Branches 14990 15252 +262
=============================================
- Hits 62115 35118 -26997
- Misses 31299 62147 +30848
+ Partials 4585 3008 -1577
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
xiangfu0
added
the
release-notes
Jackie-Jiang
added
the
Configuration
|
Note that since we enabled the row time value check by default, it might slow down the ingestion for existing use cases. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
_skipTimeValueCheckin SegmentGeneratorConfig.Release Notes
_continueOnErrorwhich if set to true will let Pinot skip any row indexing error and move on to the next row._rowTimeValueCheckwhich if set to true will let Pinot check the time value for each row and replace the value with null if the value is invalid._segmentTimeValueCheckwhich if set to false will let Pinot bypass any segment metadata level time validation.