roachtest: validate-system-schema works in shared-process deployments

[renatolabs]

This commit updates the
acceptance/validate-system-schema-after-version-upgrade roachtest so
that it can now work on multi-tenant (shared-process) deployments.

In multi-tenant deployments, we also capture the system schema on the
tenant and make the same assertion that the schema should match
whether we bootstrapped on a version or upgraded to it. In addition,
we also verify that the system schema is the same on the system and
non-system tenants.

Informs: #127378

Release note: None

[cockroach-teamcity]

This change is 

[renatolabs]

This is failing on some shared-process runs. Some system tables are missing the exclude_data_from_backup bit on secondary tenants. This seems related to #120144.

The failure can be reproduced with seed -4961013970065068305 on this branch. Output:

After upgrading, `USE system; SHOW CREATE ALL TABLES;` does not match expected output after version upgrade for secondary tenant.
Diff:
@@ -89,11 +89,11 @@
        version INT8 NOT NULL,
        sql_instance_id INT8 NOT NULL,
        session_id BYTES NOT NULL,
        crdb_region BYTES NOT NULL,
        CONSTRAINT "primary" PRIMARY KEY (crdb_region ASC, desc_id ASC, version ASC, session_id ASC)
-) WITH (exclude_data_from_backup = true);
+);
 CREATE TABLE public.locations (
        "localityKey" STRING NOT NULL,
        "localityValue" STRING NOT NULL,
        latitude DECIMAL(18,15) NOT NULL,
        longitude DECIMAL(18,15) NOT NULL,
@@ -184,11 +184,11 @@
        config STRING NOT NULL,
        report_id INT8 NOT NULL,
        violation_start TIMESTAMPTZ NULL,
        violating_ranges INT8 NOT NULL,
        CONSTRAINT "primary" PRIMARY KEY (zone_id ASC, subzone_id ASC, type ASC, config ASC)
-) WITH (exclude_data_from_backup = true);
+);
 CREATE TABLE public.replication_critical_localities (
        zone_id INT8 NOT NULL,
        subzone_id INT8 NOT NULL,
        locality STRING NOT NULL,
        report_id INT8 NOT NULL,
@@ -202,11 +202,11 @@
        total_ranges INT8 NOT NULL,
        unavailable_ranges INT8 NOT NULL,
        under_replicated_ranges INT8 NOT NULL,
        over_replicated_ranges INT8 NOT NULL,
        CONSTRAINT "primary" PRIMARY KEY (zone_id ASC, subzone_id ASC)
-) WITH (exclude_data_from_backup = true);
+);
 CREATE TABLE public.reports_meta (
        id INT8 NOT NULL,
        "generated" TIMESTAMPTZ NOT NULL,
        CONSTRAINT "primary" PRIMARY KEY (id ASC)
 );

@cockroachdb/sql-foundations Is this test expected to pass on secondary tenants?

[rafiss]

Yes, it is expected to pass, so I think your change surfaced a bug. Luckily, it's low impact, since it just means that certain system tables for shared-process tenants will not be excluded from backup. They were excluded in #120144 just as an optimization to avoid setting protected timestamps on them.

The mistake in #120144 was that the following block only appeared in the pkg/upgrade/upgrades/v24_1_system_database.go upgrade gated behind a if deps.Codec.ForSystemTenant() condition. It should have been done unconditionally.

	// Mark the tables with low gc.ttl as excluded from backup.
		if err := deps.DB.DescsTxn(ctx, func(ctx context.Context, txn descs.Txn) error {
			for _, tabName := range []catconstants.SystemTableName{
				catconstants.ReplicationConstraintStatsTableName,
				catconstants.ReplicationStatsTableName,
				catconstants.TenantUsageTableName,
				catconstants.LeaseTableName,
				catconstants.SpanConfigurationsTableName,
			} {
				if _, err := txn.ExecEx(ctx,
					"mark-table-excluded-from-backup",
					txn.KV(),
					sessiondata.NodeUserSessionDataOverride,
					"ALTER TABLE system.public."+string(tabName)+" SET (exclude_data_from_backup = true)",
				); err != nil {
					return err
				}
			}
			return nil
		}); err != nil {
			return err
		}

To fix it, we can add another upgrade that does the above for non-system tenants.

[srosenberg]

Nice!

[renatolabs]

TFTR!

bors r=srosenberg

[craig]

Build succeeded:

• lint
• macos_amd64_build
• acceptance
• linux_amd64_build
• docker_image_amd64
• macos_arm64_build
• check_generated_code
• linux_amd64_fips_build
• linux_arm64_build
• windows_build
• examples_orms
• unit_tests
• local_roachtest
• local_roachtest_fips

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error setting reviewers, but backport branch blathers/backport-release-23.2-129485 is ready: POST https://api.github.com/repos/cockroachdb/cockroach/pulls/129799/requested_reviewers: 422 Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the cockroachdb/cockroach repository. []

Backport to branch 23.2.x failed. See errors above.

---

error setting reviewers, but backport branch blathers/backport-release-24.1-129485 is ready: POST https://api.github.com/repos/cockroachdb/cockroach/pulls/129800/requested_reviewers: 422 Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the cockroachdb/cockroach repository. []

Backport to branch 24.1.x failed. See errors above.

---

error setting reviewers, but backport branch blathers/backport-release-24.2-129485 is ready: POST https://api.github.com/repos/cockroachdb/cockroach/pulls/129801/requested_reviewers: 422 Reviews may only be requested from collaborators. One or more of the teams you specified is not a collaborator of the cockroachdb/cockroach repository. []

Backport to branch 24.2.x failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}