Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: run tests against larger clusters #26978

Closed
tbg opened this issue Jun 26, 2018 · 1 comment
Closed

roachtest: run tests against larger clusters #26978

tbg opened this issue Jun 26, 2018 · 1 comment
Labels
A-kv-client Relating to the KV client and the KV interface. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@tbg
Copy link
Member

tbg commented Jun 26, 2018

A user has privately shared multiple issues found working with generally ~32 node clusters, including stuck requests during IMPORT and minutes of failed liveness heartbeats (the first likely and the second definitely related to a GC'able replica that isn't getting removed in due time).

While we generally have some theory and understanding about these issues, we should be more proactive about running into them ourselves. A few tests that run heavy rebalancing plus chaos along with regular activity (that is somewhat latency-sensitive, i.e. we would notice if it got stuck for 30+s) on most ranges should be sufficient.

@tbg tbg added the A-kv-client Relating to the KV client and the KV interface. label Jun 26, 2018
@tbg tbg added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Jul 22, 2018
@tbg tbg added this to the 2.1 milestone Jul 22, 2018
@petermattis petermattis removed this from the 2.1 milestone Oct 5, 2018
@tbg
Copy link
Member Author

tbg commented Dec 4, 2018

We're running various imports/restores against clusters of size 8 and 32, and that has turned up a number of issues that are now fixed (#31409). There's always more to do, but this issue is too general.

@tbg tbg closed this as completed Dec 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-kv-client Relating to the KV client and the KV interface. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Development

No branches or pull requests

2 participants