cli,clusterversion: high cluster version count breaks debug zips

[irfansharif]

Describe the problem

#77337 introduces two cluster versions, and observed in CI that doing so partially breaks debug zip collection of the system tables.

To Reproduce

To repro, apply:

diff --git i/pkg/clusterversion/cockroach_versions.go w/pkg/clusterversion/cockroach_versions.go
index 4bedf909ef..7945ca077e 100644
--- i/pkg/clusterversion/cockroach_versions.go
+++ w/pkg/clusterversion/cockroach_versions.go
@@ -512,7 +512,7 @@ var versionsSingleton = keyedVersions{
 	},
 	{
 		Key:     RowLevelTTL,
-		Version: roachpb.Version{Major: 21, Minor: 2, Internal: 88},
+		Version: roachpb.Version{Major: 21, Minor: 2, Internal: 92},
 	},
 
 	// *************************************************

Spin up a cockroach shell and run the following:

> SELECT *, to_hex(value::bytes) as hex_value FROM system.settings LIMIT 5;
ERROR: bytea encoded value ends with escape character

This is the query we use to populate debug zips:

cockroach/pkg/cli/zip.go

Line 53 in e097fc4

"system.settings": "SELECT *, to_hex(value::bytes) as hex_value FROM system.settings",

Expected behavior

For the above not to fail. Because it does, debug zips will fail to capture system.settings data.

+cc @ajwerner who touched this last in #70498. I'm not sure who a better owner is. Server?

Jira issue: CRDB-13662

[blathers-crl]

Hi @irfansharif, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[knz]

When someone works on this, I'll gladly review the PR.

[ajwerner]

The root cause here is interesting, and sort of terrible: we are storing bytes data in a string column in this system table.

[knz]

@ajwerner so this issue is not really obs inf then, it's more of a SQL / shared systems thing?

[ajwerner]

I think there's two facets here. We're using a STRING column but storing arbitrary bytes in it, that seems to cause problems when casting that field to bytea. We probably want a short-term fix to find a way to directly encode the underlying bytes to hex. I don't know how to represent that in SQL.

1. Decide whether we want to do something about the fact that we're storing raw bytes in a STRING column, which should be a shared systems thing
2. Work around the fact that we're doing this to make debug zip generation robust to the current state of the world.

I think the second issue does belong here.

[knz]

We probably want a short-term fix to find a way to directly encode the underlying bytes to hex. I don't know how to represent that in SQL

Ok, that seems hard. I can see two different ways:

• don't do it server-side via to_hex, instead do it client-side in the zip code.
• cast the string to JSONB, then hex-encode the entire JSONB value

[knz]

cc @rimadeodhar for triage

[ajwerner]

I could see us adding a new builtin that just interprets a STRING as bytes directly.

[knz]

I'd like a solution that's guaranteed to work on previous-version clusters too

[dhartunian]

to_hex can take a string argument. Is there a reason we can't do that here?

root@:26257/defaultdb> SELECT *, to_hex(value::bytes) as hex_value FROM system.settings;
              name              |                value                 |        lastUpdated         | valueType |                                hex_value
--------------------------------+--------------------------------------+----------------------------+-----------+---------------------------------------------------------------------------
  admission.kv.enabled          | true                                 | 2022-02-28 15:22:45.659777 | b         | 74727565
  cluster.secret                | 51358315-f723-47c8-bc28-b1e0da6254ff | 2022-02-03 20:18:51.936263 | s         | 35313335383331352d663732332d343763382d626332382d623165306461363235346666
  diagnostics.reporting.enabled | true                                 | 2022-02-03 20:18:51.772724 | b         | 74727565
(3 rows)
(error encountered after some results were delivered)
ERROR: bytea encoded value ends with escape character
SQLSTATE: 22025

root@:26257/defaultdb> SELECT *, to_hex(value) as hex_value FROM system.settings;
              name              |                value                 |        lastUpdated         | valueType |                                hex_value
--------------------------------+--------------------------------------+----------------------------+-----------+---------------------------------------------------------------------------
  admission.kv.enabled          | true                                 | 2022-02-28 15:22:45.659777 | b         | 74727565
  cluster.secret                | 51358315-f723-47c8-bc28-b1e0da6254ff | 2022-02-03 20:18:51.936263 | s         | 35313335383331352d663732332d343763382d626332382d623165306461363235346666
  diagnostics.reporting.enabled | true                                 | 2022-02-03 20:18:51.772724 | b         | 74727565
  version                       | \x12\b\b\x15\x10\x02\x18\x00 \\      | 2022-03-10 19:51:48.770627 | m         | 1208081510021800205c
(4 rows)


Time: 1ms total (execution 1ms / network 0ms)

[ajwerner]

🙌 that's the ticket!