Enable sse2 for CSV parsing. #2977
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| && *next_pos != delimiter && *next_pos != '\r' && *next_pos != '\n') /// NOTE You can make a SIMD version. | ||
| ++next_pos; | ||
|
|
||
| [&]() { |
There was a problem hiding this comment.
Why lambda? Just code block {...} is Ok.
There was a problem hiding this comment.
There is a early return
|
|
||
| [&]() { | ||
| #if __SSE2__ | ||
| auto rc = _mm_set1_epi8('\r'); |
There was a problem hiding this comment.
Shouldn't we include the corresponding ...intrin.h?
dbms/src/IO/ReadHelpers.cpp
Outdated
| __v2du a = reinterpret_cast<__v2du>(_mm_cmpeq_epi8(bytes, rc)); | ||
| __v2du b = reinterpret_cast<__v2du>(_mm_cmpeq_epi8(bytes, nc)); | ||
| __v2du c = reinterpret_cast<__v2du>(_mm_cmpeq_epi8(bytes, dc)); | ||
| __m128i eq = reinterpret_cast<__m128i>(a | b | c); |
There was a problem hiding this comment.
Ok. But I don't understand, how is it better than writing two _mm_or_si128 instead?
Isn't the __v2du less portable or less documented?
There was a problem hiding this comment.
Hmm, lemme check the output assembly. gcc folks haven't answered back yet.
There was a problem hiding this comment.
They're exactly the same. I'll change to _mm_or_si128
| && *next_pos != delimiter && *next_pos != '\r' && *next_pos != '\n') /// NOTE You can make a SIMD version. | ||
| ++next_pos; | ||
|
|
||
| [&]() { |
Testing data
```
select 'aaaaaaaa,bbbbbbbb,cccccccc,dddddddd,eeeeeeee,ffffffff,gggg,hhh' from numbers(3000000) into outfile '/tmp/test.csv'
```
Testing command
```
echo "select count() from file('/tmp/test.csv', CSV, 'a String, b String, c String, d String, e String, f String, g String, h String') where not ignore(e)" | clickhouse-benchmark
```
Before
```
QPS: 1.317, RPS: 3949749.687, MiB/s: 478.380, result RPS: 1.317, result MiB/s: 0.000.
0.000% 0.704 sec.
10.000% 0.712 sec.
20.000% 0.718 sec.
30.000% 0.726 sec.
40.000% 0.739 sec.
50.000% 0.754 sec.
60.000% 0.770 sec.
70.000% 0.788 sec.
80.000% 0.798 sec.
90.000% 0.815 sec.
95.000% 0.826 sec.
99.000% 0.850 sec.
99.900% 0.857 sec.
99.990% 0.858 sec.
```
After
```
QPS: 1.533, RPS: 4598308.336, MiB/s: 556.932, result RPS: 1.533, result MiB/s: 0.000.
0.000% 0.626 sec.
10.000% 0.635 sec.
20.000% 0.639 sec.
30.000% 0.642 sec.
40.000% 0.643 sec.
50.000% 0.645 sec.
60.000% 0.649 sec.
70.000% 0.652 sec.
80.000% 0.658 sec.
90.000% 0.682 sec.
95.000% 0.710 sec.
99.000% 0.727 sec.
99.900% 0.733 sec.
99.990% 0.734 sec.
```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Testing data
Testing command
Before
After
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en