Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve response processing #941

Merged
merged 2 commits into from
Mar 30, 2020
Merged

Improve response processing #941

merged 2 commits into from
Mar 30, 2020

Conversation

danielmitterdorfer
Copy link
Member

With this commit we improve response processing speed by taking the following
measures:

  1. We use the raw bytes for JSON parsing instead of first converting them into a
    string.
  2. We expose an additional option for scroll queries to disable HTTP response
    compression as we have seen that this can cause significant overhead for very
    large responses (large documents). The default is (still) to have response
    compression enabled.

Our experiments have shown the following changes to the 50th percentile service
time for the PMC track:

  • Baseline: 3203 ms
  • With measure 1: 3014 ms
  • With measure 1 and measure 2: 774 ms

Relates #935

@danielmitterdorfer danielmitterdorfer added enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc. labels Mar 30, 2020
@danielmitterdorfer danielmitterdorfer added this to the 1.5.0 milestone Mar 30, 2020
@danielmitterdorfer danielmitterdorfer self-assigned this Mar 30, 2020
danielmitterdorfer added a commit to danielmitterdorfer/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio
623426a 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
@danielmitterdorfer danielmitterdorfer mentioned this pull request Mar 30, 2020
Merged
dliappis
dliappis approved these changes Mar 30, 2020
View reviewed changes
Copy link
Contributor

@dliappis dliappis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM the measures taken here make sense to me.

danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio (#108)
fc85b9d 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio (#108)
e6e8e24 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio (#108)
d9b50ce 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio (#108)
8ce9f43 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio (#108)
6cf0062 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio (#108)
2ee4d26 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
danielmitterdorfer added a commit to elastic/rally-tracks that referenced this pull request Mar 30, 2020
@danielmitterdorfer
Adapt for asyncio (#108)
171563a 
With this commit we change some of our tracks to make sure nightly
Elasticsearch benchmarks continue to run smoothly with the new asyncio
load generator (see elastic/rally#935):

* We lower the target throughput of some queries
* We disable HTTP response compression for PMC scroll queries

Relates elastic/rally#935
Relates elastic/rally#941
@danielmitterdorfer danielmitterdorfer merged commit 9979f8a into elastic:master Mar 30, 2020
@danielmitterdorfer danielmitterdorfer deleted the faster-response-processing branch March 30, 2020 11:49
@danielmitterdorfer
Copy link
Member Author

Thanks for the review!

danielmitterdorfer added a commit to danielmitterdorfer/rally that referenced this pull request Mar 31, 2020
@danielmitterdorfer
Honor response-compression-enabled parameter
35dd4f9 
With this commit we ensure that the search parameter source passes the
specified parameter `response-compression-enabled` to the runner.

Relates elastic#941
@danielmitterdorfer danielmitterdorfer mentioned this pull request Mar 31, 2020
Merged
danielmitterdorfer added a commit that referenced this pull request Mar 31, 2020
@danielmitterdorfer
Honor response-compression-enabled parameter (#942)
c3caddd 
With this commit we ensure that the search parameter source passes the
specified parameter `response-compression-enabled` to the runner.

Relates #941
danielmitterdorfer added a commit to danielmitterdorfer/rally that referenced this pull request Apr 3, 2020
@danielmitterdorfer
Allow to disable HTTP compression for all queries
2a1fa1d 
With this commit we expand the support added in elastic#941 so that not only
scroll queries can take advantage but also regular queries.

Relates elastic#941
@danielmitterdorfer danielmitterdorfer mentioned this pull request Apr 3, 2020
Merged
danielmitterdorfer added a commit that referenced this pull request Apr 7, 2020
@danielmitterdorfer
Allow to disable HTTP compression for all queries (#952)
b120fbd 
With this commit we expand the support added in #941 so that not only
scroll queries can take advantage but also regular queries.

Relates #941
@dliappis dliappis mentioned this pull request Aug 10, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dliappis dliappis dliappis approved these changes

Assignees

@danielmitterdorfer danielmitterdorfer

Labels
enhancement Improves the status quo :Load Driver Changes that affect the core of the load driver such as scheduling, the measurement approach etc.
Projects
None yet
Milestone
2.0.0
Development

Successfully merging this pull request may close these issues.
2 participants
@danielmitterdorfer @dliappis