Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcdserver: prune revisions before send backend readTx when rangekeys #12305

Conversation

xrmzju
Copy link

@xrmzju xrmzju commented Sep 16, 2020

@xrmzju
Copy link
Author

xrmzju commented Sep 16, 2020

this improves performance tremendously for request WithLastRev() or WithFristRev() option when key range is big
without this pr:

[root@edd2693ca788 /]# time ./etcdctl  get --prefix=true /default/ --last-rev=true --keys-only -wjson --command-timeout=60s
{"header":{"cluster_id":11136150232382961551,"member_id":4898650950304686290,"revision":1337662,"raft_term":23},"kvs":[{"key":"L2RlZmF1bHQvQjFDVFl5d0ZxSVFpb2ZHMXBPcXpiVGtBbWVKTFQyZGc=","create_revision":1337662,"mod_revision":1337662,"version":1}],"more":true,"count":1317250}

real	0m35.135s
user	0m0.031s
sys	0m0.022s

with this pr



[root@edd2693ca788 /]# time ./etcdctl   get --prefix=true /default/ --last-rev=true --keys-only -wjson
{"header":{"cluster_id":11136150232382961551,"member_id":4898650950304686290,"revision":1337662,"raft_term":22},"kvs":[{"key":"L2RlZmF1bHQvQjFDVFl5d0ZxSVFpb2ZHMXBPcXpiVGtBbWVKTFQyZGc=","create_revision":1337662,"mod_revision":1337662,"version":1}],"more":true,"count":1317250}

real	0m0.892s
user	0m0.027s
sys	0m0.023s

@xrmzju
Copy link
Author

xrmzju commented Sep 16, 2020

@xiang90 @gyuho @jpbetz @jingyih @spzala please take a look. thx.

@xrmzju xrmzju force-pushed the improve-range-performance-with-filterd-revision branch 2 times, most recently from 8f7987f to 6049144 Compare September 16, 2020 12:55
@xrmzju xrmzju changed the title prune revisions before send backend readTx when rangekeys etcdserver: prune revisions before send backend readTx when rangekeys Sep 16, 2020
@xrmzju
Copy link
Author

xrmzju commented Sep 16, 2020

i don't see why the ci build failed ,can somebody help me...

@xrmzju xrmzju force-pushed the improve-range-performance-with-filterd-revision branch from 6049144 to fcbd007 Compare September 16, 2020 14:24
@xrmzju xrmzju force-pushed the improve-range-performance-with-filterd-revision branch 2 times, most recently from 4df803c to 3378fa0 Compare September 17, 2020 09:40
@tangcong
Copy link
Contributor

=== RUN TestV3RangeRequest
TestV3RangeRequest: v3_grpc_test.go:1514: #4.3: key[0]. got = a, want = c,
--- FAIL: TestV3RangeRequest (1.12s)

@xrmzju can you take a look at this failed test case?

@xrmzju xrmzju force-pushed the improve-range-performance-with-filterd-revision branch from 3378fa0 to 6648f35 Compare September 17, 2020 14:47
@xrmzju
Copy link
Author

xrmzju commented Sep 17, 2020

=== RUN TestV3RangeRequest
TestV3RangeRequest: v3_grpc_test.go:1514: #4.3: key[0]. got = a, want = c,
--- FAIL: TestV3RangeRequest (1.12s)

@xrmzju can you take a look at this failed test case?

fixed

@xrmzju xrmzju force-pushed the improve-range-performance-with-filterd-revision branch from 6648f35 to d7c3bd2 Compare September 17, 2020 14:59
@xrmzju
Copy link
Author

xrmzju commented Sep 21, 2020

if u guys have any other feedbacks, please let me know, many thanks! @gyuho @jpbetz @jingyih @tangcong @cfc4n

@stale
Copy link

stale bot commented Dec 20, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Dec 20, 2020
@stale stale bot closed this Jan 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging this pull request may close these issues.

storeTxnRead should filter revisions in mvcc before send to backend
3 participants