workload: add follower read support

[andrewbaptist]

The performance of the system can be different if there are reads as of a system time. This PR adds the ability to issue follower reads.

Epic: none

Release note (cli change): Add --follower-read-percent to the workload kv run.

[cockroach-teamcity]

This change is 

[andrewbaptist]

This is part of the testing effort for #112351. It allows running follower reads for KV workloads.

[andrewbaptist]

@kvoli This can't merge yet as this depends on the fix for follower reads (otherwise this test will fail). However, I wanted to have you review these changes as they are in a test you have been mostly involved in.

[nvanbenschoten]

Reviewed 1 of 1 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @andrewbaptist, @herkolategan, @kvoli, @msbutler, and @srosenberg)

---

pkg/cmd/roachtest/tests/kv.go line 542 at r2 (raw file):

			// Don't connect to the node we are going to shut down.
			dbs := make([]*gosql.DB, nodes-1)
			for i := 0; i < nodes-1; i++ {

nit: for i := range dbs {

---

pkg/cmd/roachtest/tests/kv.go line 615 at r2 (raw file):

			t.Status("gracefully draining and restarting nodes")
			// Gracefully shut down the third node, let the cluster run for a
			// while, then restart it. Then repeat for good measure.

s/Then repeat/Repeat for a total of 3 times/

To make it more clear where the 3 is coming from below.

---

pkg/cmd/roachtest/tests/kv.go line 637 at r2 (raw file):

				// drop all packets in both directions.
				if !c.IsLocal() {
					c.Run(ctx, c.Node(nodes), `sudo iptables -A INPUT -p tcp --dport 26257 -j DROP`)

Nice idea.

---

pkg/workload/kv/kv.go line 99 at r1 (raw file):

	insertCount                          int
	useBackgroundTxnQoS                  bool
	followerReadPercent                  int

nit: move this right below readPercent.

---

pkg/workload/kv/kv.go line 185 at r1 (raw file):

		g.flags.BoolVar(&g.useBackgroundTxnQoS, `background-qos`, false,
			`Set default_transaction_quality_of_service session variable to "background".`)
		g.flags.IntVar(&g.followerReadPercent, `follower-read-percent`, 0,

nit: move this below g.flags.IntVar(&g.readPercent...

---

pkg/workload/kv/kv.go line 185 at r1 (raw file):

		g.flags.BoolVar(&g.useBackgroundTxnQoS, `background-qos`, false,
			`Set default_transaction_quality_of_service session variable to "background".`)
		g.flags.IntVar(&g.followerReadPercent, `follower-read-percent`, 0,

It was previously possible to accomplish this by adding the default_transaction_use_follower_reads session variable to the db URL that you pass to KV, but this is much nicer and also allows for a fraction of the read to be from followers.

---

pkg/workload/kv/kv.go line 601 at r1 (raw file):

		readStmt := o.readStmt

		if o.config.followerReadPercent > 0 &&

nit: do we need the o.config.followerReadPercent > 0 condition?

[andrewbaptist]

TFTR - I updated the code, but am only going to merge the first commit (change workload to support follower reads) here once approved, and then move the other commit to the branch which has the fix for follower read stopping.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @herkolategan, @kvoli, @msbutler, @nvanbenschoten, and @srosenberg)

---

pkg/cmd/roachtest/tests/kv.go line 542 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

nit: for i := range dbs {

Done

---

pkg/cmd/roachtest/tests/kv.go line 615 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

s/Then repeat/Repeat for a total of 3 times/

To make it more clear where the 3 is coming from below.

Done

---

pkg/cmd/roachtest/tests/kv.go line 637 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Nice idea.

Thanks - I realized it wasn't reproducing reliably with "clean" shutdown.

---

pkg/workload/kv/kv.go line 99 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

nit: move this right below readPercent.

Done

---

pkg/workload/kv/kv.go line 185 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

nit: move this below g.flags.IntVar(&g.readPercent...

Done

---

pkg/workload/kv/kv.go line 185 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

It was previously possible to accomplish this by adding the default_transaction_use_follower_reads session variable to the db URL that you pass to KV, but this is much nicer and also allows for a fraction of the read to be from followers.

Right - I had considered that previously, but I wanted to make sure that the test at least covered both types of reads since the cause for slowdowns might have been different for each.

---

pkg/workload/kv/kv.go line 601 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

nit: do we need the o.config.followerReadPercent > 0 condition?

No - removed to make it more clear.

[kvoli]

First commit

Reviewed 2 of 2 files at r3.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @andrewbaptist, @herkolategan, @msbutler, @nvanbenschoten, and @srosenberg)

---

-- commits line 5 at r1:
nit: could you mention adding follower read support for the kv workload specifically.

[andrewbaptist]

bors r=kvoli

TFTR!
I pulled off the second commit so I can add the changes to the workload tool first prior to the follower read changes being made. I moved the second commit to: #112926.

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)