Merge #63472

63472: roachprod: use exit-on-error:false for crdb log cfg r=ajwerner a=tbg See #62763. We seem to frequently miss the runtime errors resulting from out-of-memory conditions in the stderr logs. We don't understand exactly why yet, but it is very likely that with `exit-on-error` (which is true by default) we are hitting errors outputting to the sink which then kill the process before the runtime errors bubble up. While we develop a proper fix, avoid the problematic configuration on roachprod clusters, which notably includes roachtests. Release note: None Co-authored-by: Tobias Grieger <[email protected]>
cockroachdb · Apr 14, 2021 · b2a6c7d · b2a6c7d
2 parents dc7f6f0 + 98ee488
commit b2a6c7d
Showing 1 changed file with 4 additions and 1 deletion.
diff --git a/pkg/cmd/roachprod/install/cockroach.go b/pkg/cmd/roachprod/install/cockroach.go
@@ -457,7 +457,10 @@ func (h *crdbInstallHelper) generateStartArgs(
 	}
 
 	logDir := h.c.Impl.LogDir(h.c, nodes[nodeIdx])
-	args = append(args, "--log-dir="+logDir)
+	if vers.AtLeast(version.MustParse("v21.1.0")) {
+		// Specify exit-on-error=false to work around #62763.
+		args = append(args, `--log "file-defaults: {dir: '`+logDir+`', exit-on-error: false}"`)
+	}
 
 	if vers.AtLeast(version.MustParse("v1.1.0")) {
 		cache := 25