Add semi-sync monitor to unblock primaries blocked on semi-sync ACKs #17763

GuptaManan100 · 2025-02-12T15:01:48Z

Description

This PR introduces a new component to the vttablet binary to monitor the semi-sync status of primary vttablets. We've observed cases where a brief network disruption can cause the primary to get stuck indefinitely waiting for semi-sync ACKs. In rare scenarios, this can block reparent operations and render the primary unresponsive. More information can be found in the issues #17709 and #17749.

To address this, the new component continuously monitors the semi-sync status. If the primary becomes stuck on semi-sync ACKs, it generates writes to unblock it. If this fails, VTOrc is notified of the issue and initiates an emergency reparent operation.

A metric for the number of outstanding writes from the semi-sync monitor has also been added. Unfortunately, it's not easy to reproduce the problem in an end-to-end test since it requires setting port forward rules (on Mac) or iptable changes (on Linux), both of which require sudo access. So I've added a test but that test doesn't run on CI. It can only be run locally and the password for the root user has to entered when prompted for it.

The semi-sync monitor is used such that the users who aren't running semi-sync, will not see the monitor open at all. It will only run when semi-sync on the primary is turned on.

Related Issue(s)

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required

Deployment Notes

Signed-off-by: Manan Gupta <[email protected]>

vitess-bot · 2025-02-12T15:01:52Z

codecov · 2025-02-12T16:01:13Z

Codecov Report

Attention: Patch coverage is 82.13058% with 52 lines in your changes missing coverage. Please review.

Project coverage is 67.48%. Comparing base (67d081a) to head (ee098c6).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
.../vttablet/tabletmanager/semisyncmonitor/monitor.go	89.65%	24 Missing ⚠️
go/vt/vttablet/tabletserver/tabletenv/config.go	44.00%	14 Missing ⚠️
go/vt/vttablet/tabletmanager/rpc_replication.go	15.38%	11 Missing ⚠️
go/vt/vtorc/inst/instance_dao.go	75.00%	1 Missing ⚠️
go/vt/vtorc/logic/topology_recovery.go	0.00%	1 Missing ⚠️
go/vt/wrangler/testlib/fake_tablet.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17763      +/-   ##
==========================================
+ Coverage   67.41%   67.48%   +0.06%     
==========================================
  Files        1592     1593       +1     
  Lines      258024   258473     +449     
==========================================
+ Hits       173948   174424     +476     
+ Misses      84076    84049      -27

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Manan Gupta <[email protected]>

go/vt/vttablet/tabletserver/semisyncwatcher/watcher.go

mattlord · 2025-02-13T15:44:24Z

go/vt/vttablet/tabletserver/semisyncwatcher/watcher.go

+func (w *Watcher) stillBlocked() bool {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	return w.isOpen && w.isBlocked
+}
+
+// checkAndSetIsWriting checks if the watcher is already writing to the DB.
+// If it is not, then it sets the isWriting field and signals the caller.
+func (w *Watcher) checkAndSetIsWriting() bool {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	if w.isWriting {
+		return false
+	}
+	w.isWriting = true
+	return true
+}


I don't think this affects the query hot path, right? If it does, then it might be worth e.g. using 1 byte for the status and using bits in there for isWriting, isBlocked, isOpen etc. so that we can use atomics for reading them, CAS for optional changes, etc. If nothing else, it's probably worth moving these to atomic.Bool so that e.g. checkAndSetIsWriting can be one atomic call:

return w.isWriting(false, true)

It makes the code simpler, clearer, and less contentious / efficient.

I don't think performance is too much of a concern, but the usage of having multiple bool fields behind a mutex vs atomic.Bool I think becomes a matter of preference. I for one, like to have the former because that means that only one boolean value transitions at a point in time, but with atomic bool values it can change even when you've just read that value.

go/vt/vttablet/tabletserver/semisyncwatcher/watcher.go

Signed-off-by: Manan Gupta <[email protected]>

…writes Signed-off-by: Manan Gupta <[email protected]>

…c is blocked on the primary Signed-off-by: Manan Gupta <[email protected]>

…d for too long and a add a few logs Signed-off-by: Manan Gupta <[email protected]>

Signed-off-by: Manan Gupta <[email protected]>

…ger to only start it when required Signed-off-by: Manan Gupta <[email protected]>

Signed-off-by: Manan Gupta <[email protected]>

…ake tablet Signed-off-by: Manan Gupta <[email protected]>

… is blocked Signed-off-by: Manan Gupta <[email protected]>

Signed-off-by: Manan Gupta <[email protected]>

mattlord

This is looking good! Just had some questions/comments. Let me know what you think.

go/test/endtoend/reparent/newfeaturetest/reparent_test.go

go/vt/vttablet/tabletserver/tabletenv/config.go

go/vt/vttablet/tabletmanager/rpc_replication_test.go