Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Parquet] Performance improvement by checking column dictionary earlier #2676

Merged
merged 15 commits into from
Jul 20, 2023

Conversation

mdisibio
Copy link
Contributor

@mdisibio mdisibio commented Jul 19, 2023

What this PR does:
Checks column dictionaries as early as possible in KeepColumnChunk so that we can skip over them properly. Previously we had to wait for the first page because of the library design, and even though we short-circuited some of the logic, we still had to perform all of the I/o and decoding. This improvement mostly applies to needle-in-the-haystack or low cardinality values, where the skip occurs. Benchmarks below. In the situations where MB/s went down, it's because the I/O went down faster than cpu.

This new way of checking the dictionary also benefits SearchTagValues. I know this will conflict with the vParquet3 PR so we can revert it if we want, but there is still at least the 1-line change to the Predicate.KeepColumnChunk method signature.

name                                                 old time/op    new time/op    delta
BackendBlockTraceQL/spanAttNameNoMatch-12              4.78ms ± 1%    1.65ms ± 0%  -65.51%  (p=0.000 n=9+10)
BackendBlockTraceQL/spanAttValNoMatch-12               4.66ms ± 1%    1.64ms ± 4%  -64.88%  (p=0.000 n=8+10)
BackendBlockTraceQL/spanAttValMatch-12                 4.67ms ± 1%    1.61ms ± 1%  -65.49%  (p=0.000 n=9+8)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12         2.54ms ± 3%    1.51ms ± 1%  -40.70%  (p=0.000 n=9+9)
BackendBlockTraceQL/spanAttIntrinsicMatch-12           53.3ms ± 3%    52.7ms ± 1%   -1.16%  (p=0.019 n=9+9)
BackendBlockTraceQL/spanAttIntrinsicMatchRegex-12      80.0ms ± 5%    79.7ms ± 3%     ~     (p=0.912 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicNoMatchRegex-12    2.68ms ± 1%    1.69ms ± 2%  -36.88%  (p=0.000 n=8+10)
BackendBlockTraceQL/resourceAttNameNoMatch-12          2.90ms ± 1%    1.45ms ± 2%  -49.92%  (p=0.000 n=8+10)
BackendBlockTraceQL/resourceAttValNoMatch-12           9.93ms ± 1%    6.93ms ± 2%  -30.28%  (p=0.000 n=9+9)
BackendBlockTraceQL/resourceAttValMatch-12              376ms ± 2%     360ms ± 2%   -4.20%  (p=0.000 n=9+10)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12     1.65ms ± 4%    1.40ms ± 2%  -14.86%  (p=0.000 n=10+10)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12       22.5ms ± 1%    21.2ms ± 1%   -5.60%  (p=0.000 n=9+10)
BackendBlockTraceQL/unscopedMatch-12                    342ms ± 2%     319ms ± 1%   -6.91%  (p=0.000 n=10+9)
BackendBlockTraceQL/mixedNameNoMatch-12                 407ms ± 2%     377ms ± 1%   -7.53%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValNoMatch-12                  411ms ± 1%     389ms ± 3%   -5.44%  (p=0.000 n=9+10)
BackendBlockTraceQL/mixedValMixedMatchAnd-12           2.92ms ± 1%    1.46ms ± 2%  -49.98%  (p=0.000 n=9+10)
BackendBlockTraceQL/mixedValMixedMatchOr-12             320ms ± 1%     309ms ± 3%   -3.68%  (p=0.000 n=9+9)
BackendBlockTraceQL/mixedValBothMatch-12               1.59ms ± 2%    1.39ms ± 1%  -12.52%  (p=0.000 n=10+10)

name                                                 old speed      new speed      delta
BackendBlockTraceQL/spanAttNameNoMatch-12             672MB/s ± 1%  1310MB/s ± 0%  +94.94%  (p=0.000 n=9+10)
BackendBlockTraceQL/spanAttValNoMatch-12              689MB/s ± 1%  1320MB/s ± 3%  +91.49%  (p=0.000 n=8+10)
BackendBlockTraceQL/spanAttValMatch-12                688MB/s ± 1%  1341MB/s ± 1%  +94.84%  (p=0.000 n=9+8)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12        453MB/s ± 3%   763MB/s ± 1%  +68.59%  (p=0.000 n=9+9)
BackendBlockTraceQL/spanAttIntrinsicMatch-12          351MB/s ± 2%   355MB/s ± 1%   +1.16%  (p=0.019 n=9+9)
BackendBlockTraceQL/spanAttIntrinsicMatchRegex-12     234MB/s ± 4%   235MB/s ± 3%     ~     (p=0.912 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicNoMatchRegex-12   430MB/s ± 1%   681MB/s ± 2%  +58.44%  (p=0.000 n=8+10)
BackendBlockTraceQL/resourceAttNameNoMatch-12         491MB/s ± 1%   980MB/s ± 2%  +99.69%  (p=0.000 n=8+10)
BackendBlockTraceQL/resourceAttValNoMatch-12          476MB/s ± 1%   418MB/s ± 2%  -12.11%  (p=0.000 n=9+9)
BackendBlockTraceQL/resourceAttValMatch-12           64.8MB/s ± 2%  67.7MB/s ± 2%   +4.38%  (p=0.000 n=9+10)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12   80.4MB/s ± 4%  94.4MB/s ± 2%  +17.40%  (p=0.000 n=10+10)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12      792MB/s ± 1%   839MB/s ± 1%   +5.93%  (p=0.000 n=9+10)
BackendBlockTraceQL/unscopedMatch-12                 95.5MB/s ± 2%  93.9MB/s ± 1%   -1.60%  (p=0.008 n=10+9)
BackendBlockTraceQL/mixedNameNoMatch-12              50.8MB/s ± 2%  40.0MB/s ± 1%  -21.27%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValNoMatch-12               50.3MB/s ± 1%  38.8MB/s ± 3%  -22.98%  (p=0.000 n=9+10)
BackendBlockTraceQL/mixedValMixedMatchAnd-12          487MB/s ± 1%   974MB/s ± 2%  +99.93%  (p=0.000 n=9+10)
BackendBlockTraceQL/mixedValMixedMatchOr-12          76.4MB/s ± 1%  73.4MB/s ± 3%   -3.91%  (p=0.000 n=9+9)
BackendBlockTraceQL/mixedValBothMatch-12             83.2MB/s ± 2%  95.1MB/s ± 1%  +14.30%  (p=0.000 n=10+10)

name                                                 old MB_io/op   new MB_io/op   delta
BackendBlockTraceQL/spanAttNameNoMatch-12                3.21 ± 0%      2.16 ± 0%  -32.77%  (p=0.000 n=10+10)
BackendBlockTraceQL/spanAttValNoMatch-12                 3.21 ± 0%      2.16 ± 0%  -32.77%  (p=0.000 n=10+10)
BackendBlockTraceQL/spanAttValMatch-12                   3.21 ± 0%      2.16 ± 0%  -32.77%  (p=0.000 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12           1.15 ± 0%      1.15 ± 0%     ~     (all equal)
BackendBlockTraceQL/spanAttIntrinsicMatch-12             18.7 ± 0%      18.7 ± 0%     ~     (all equal)
BackendBlockTraceQL/spanAttIntrinsicMatchRegex-12        18.7 ± 0%      18.7 ± 0%     ~     (all equal)
BackendBlockTraceQL/spanAttIntrinsicNoMatchRegex-12      1.15 ± 0%      1.15 ± 0%     ~     (all equal)
BackendBlockTraceQL/resourceAttNameNoMatch-12            1.42 ± 0%      1.42 ± 0%     ~     (all equal)
BackendBlockTraceQL/resourceAttValNoMatch-12             4.72 ± 0%      2.89 ± 0%  -38.73%  (p=0.000 n=10+10)
BackendBlockTraceQL/resourceAttValMatch-12               24.4 ± 0%      24.4 ± 0%     ~     (all equal)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12       0.13 ± 0%      0.13 ± 0%     ~     (all equal)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12         17.8 ± 0%      17.8 ± 0%     ~     (all equal)
BackendBlockTraceQL/unscopedMatch-12                     32.7 ± 0%      29.9 ± 0%   -8.38%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedNameNoMatch-12                  20.7 ± 0%      15.1 ± 0%  -27.18%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValNoMatch-12                   20.7 ± 0%      15.1 ± 0%  -27.18%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValMixedMatchAnd-12             1.42 ± 0%      1.42 ± 0%     ~     (all equal)
BackendBlockTraceQL/mixedValMixedMatchOr-12              24.5 ± 0%      22.6 ± 0%   -7.48%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValBothMatch-12                 0.13 ± 0%      0.13 ± 0%     ~     (all equal)

name                                                 old alloc/op   new alloc/op   delta
BackendBlockTraceQL/spanAttNameNoMatch-12               909kB ± 9%     879kB ± 3%     ~     (p=0.075 n=10+10)
BackendBlockTraceQL/spanAttValNoMatch-12                862kB ± 7%     873kB ± 3%     ~     (p=0.684 n=10+10)
BackendBlockTraceQL/spanAttValMatch-12                  903kB ± 7%     862kB ± 3%   -4.50%  (p=0.002 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12          915kB ± 7%     902kB ± 3%     ~     (p=0.853 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicMatch-12           3.85MB ±11%    3.79MB ± 5%     ~     (p=0.971 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicMatchRegex-12      9.00MB ± 4%    8.90MB ± 5%     ~     (p=0.853 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicNoMatchRegex-12    1.02MB ± 4%    0.98MB ± 1%   -3.08%  (p=0.021 n=10+8)
BackendBlockTraceQL/resourceAttNameNoMatch-12           768kB ± 5%     778kB ± 3%     ~     (p=0.315 n=10+10)
BackendBlockTraceQL/resourceAttValNoMatch-12           7.39MB ± 2%    7.37MB ± 2%     ~     (p=0.912 n=10+10)
BackendBlockTraceQL/resourceAttValMatch-12              173MB ± 1%     173MB ± 0%     ~     (p=0.853 n=10+10)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12      781kB ± 3%     783kB ± 3%     ~     (p=0.481 n=10+10)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12        859kB ±18%     881kB ±12%     ~     (p=0.436 n=10+10)
BackendBlockTraceQL/unscopedMatch-12                   90.2MB ± 8%    87.5MB ±11%     ~     (p=0.579 n=10+10)
BackendBlockTraceQL/mixedNameNoMatch-12                90.5MB ± 8%    93.6MB ± 4%     ~     (p=0.356 n=10+9)
BackendBlockTraceQL/mixedValNoMatch-12                 95.8MB ± 4%    92.3MB ± 1%   -3.71%  (p=0.011 n=9+9)
BackendBlockTraceQL/mixedValMixedMatchAnd-12            798kB ± 8%     772kB ± 3%   -3.25%  (p=0.019 n=10+10)
BackendBlockTraceQL/mixedValMixedMatchOr-12            11.2MB ±11%    10.6MB ±13%     ~     (p=0.075 n=10+10)
BackendBlockTraceQL/mixedValBothMatch-12                804kB ± 7%     786kB ± 6%     ~     (p=0.165 n=10+10)

name                                                 old allocs/op  new allocs/op  delta
BackendBlockTraceQL/spanAttNameNoMatch-12               8.48k ± 0%     8.21k ± 0%   -3.15%  (p=0.000 n=10+10)
BackendBlockTraceQL/spanAttValNoMatch-12                8.48k ± 0%     8.21k ± 0%   -3.15%  (p=0.000 n=10+10)
BackendBlockTraceQL/spanAttValMatch-12                  8.48k ± 0%     8.21k ± 0%   -3.16%  (p=0.000 n=6+10)
BackendBlockTraceQL/spanAttIntrinsicNoMatch-12          8.28k ± 0%     8.19k ± 0%   -1.14%  (p=0.000 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicMatch-12            21.7k ± 0%     21.7k ± 0%   +0.02%  (p=0.010 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicMatchRegex-12       71.9k ± 0%     71.9k ± 0%     ~     (p=0.155 n=10+10)
BackendBlockTraceQL/spanAttIntrinsicNoMatchRegex-12     9.18k ± 0%     9.09k ± 0%   -1.03%  (p=0.000 n=10+8)
BackendBlockTraceQL/resourceAttNameNoMatch-12           8.35k ± 0%     8.20k ± 0%   -1.75%  (p=0.002 n=8+10)
BackendBlockTraceQL/resourceAttValNoMatch-12            8.51k ± 0%     8.34k ± 0%   -2.02%  (p=0.000 n=9+8)
BackendBlockTraceQL/resourceAttValMatch-12              2.02M ± 0%     2.02M ± 0%   +0.00%  (p=0.033 n=10+10)
BackendBlockTraceQL/resourceAttIntrinsicNoMatch-12      8.21k ± 0%     8.19k ± 0%   -0.24%  (p=0.000 n=10+10)
BackendBlockTraceQL/resourceAttIntrinsicMatch-12        9.29k ± 0%     9.30k ± 0%   +0.07%  (p=0.000 n=10+10)
BackendBlockTraceQL/unscopedMatch-12                    18.1k ± 0%     17.7k ± 1%   -2.19%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedNameNoMatch-12                 10.8k ± 0%      9.9k ± 0%   -8.02%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValNoMatch-12                  10.8k ± 0%      9.9k ± 0%   -8.03%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValMixedMatchAnd-12            8.36k ± 0%     8.22k ± 0%   -1.76%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValMixedMatchOr-12             22.6k ± 0%     22.3k ± 0%   -1.42%  (p=0.000 n=10+10)
BackendBlockTraceQL/mixedValBothMatch-12                8.22k ± 0%     8.20k ± 0%   -0.24%  (p=0.000 n=10+7)
name                                     old time/op    new time/op    delta
BackendBlockSearchTagValues/foo-12         32.2ms ± 3%     5.9ms ± 3%  -81.69%  (p=0.000 n=9+10)
BackendBlockSearchTagValues/http.url-12    81.2ms ± 3%    48.9ms ± 7%  -39.82%  (p=0.000 n=10+8)

name                                     old alloc/op   new alloc/op   delta
BackendBlockSearchTagValues/foo-12         1.34MB ±35%    1.16MB ± 8%     ~     (p=0.063 n=10+10)
BackendBlockSearchTagValues/http.url-12     126MB ± 1%     124MB ± 1%   -1.47%  (p=0.000 n=10+10)

name                                     old allocs/op  new allocs/op  delta
BackendBlockSearchTagValues/foo-12          11.1k ± 0%      9.0k ± 0%  -19.29%  (p=0.000 n=10+10)
BackendBlockSearchTagValues/http.url-12     97.5k ± 0%     95.3k ± 0%   -2.24%  (p=0.000 n=10+9)

Which issue(s) this PR fixes:
Fixes n/a

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Copy link
Member

@mapno mapno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool improvement

@mdisibio mdisibio merged commit 6f1c659 into grafana:main Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants