Imbalanced read results in instability of sysbench select_random_ranges/select_random_points performance (-25%~8%)

[Yui-Song]

Bug Report

Imbalanced read results in instability of sysbench select_range_point performance (-25%~8%, compared with the baseline)

What did you do?

1. Deploy a cluster
2. Turn on prepare plan cache
3. Run sysbench select_range_point

What did you expect to see?

The TPS of multiple tests are stable.

What did you see instead?

TPS：37778

TPS：54453

What version of PD are you using (pd-server -V)?

commit ae23d40

[Yui-Song]

/severity critical

[Yui-Song]

/found automation

[ti-chi-bot]

@Yui-Song: The label(s) type/performance cannot be applied, because the repository doesn't have them.

In response to this:

/type performance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@Yui-Song: The label(s) type/performance cannot be applied, because the repository doesn't have them.

In response to this:

/type performance

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[nolouch]

We need some time to fix this issue.

[Yui-Song]

/severity major

[Yui-Song]

/remove-severity critical

[ti-chi-bot]

@Yui-Song: These labels are not set on the issue: severity/critical.

In response to this:

/remove-severity critical

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[lhy1024]

After the optimization of the 6.3 and 6.4 versions, these two loads have been basically stable.

In daily run, there will be no 25% drop.

In the comparison test, select_random_ranges (6tidb3tikv), 6.4 vs. 6.3 can increase by 19%, select_random_points (3tidb3tikv) is stable in 6.3, and the performance difference between 6.3 and 6.4 is not big.

However, it is worth noting that select_random_ranges is affected by the result of splitting region. If there are still regions that are difficult to schedule after the split, the hot-region-scheduler may still not find the optimal result, so theoretically there is room for further improvement.