docs/rfcs: split up jobs system tables

[dt]

Release note: none.

Epic: none.

[cockroach-teamcity]

This change is 

[dt]

This column might not be required: I believe ideally we'd let the job itself manage its own retries and any backoff between them it thinks it should add, storing that state in its own info rows, so the backoff at the jobs system level would just be if the node running it crashed before the invoked job-specific function returned, then we should wait some interval before trying again to avoid a job of death crashing all nodes in quick succession.

[msbutler]

In the jobs_control table description above, you say, "the [job control table] is exclusively used by the jobs system itself; no user-controlled or job-controlled transactions are allowed in this table". But here, you're saying the SHOW JOBS requires reading from the control table. Are these contradictory statements, or am I confused?

[msbutler]

clarified in meeting: " user-controlled or job-controlled transactions" implies an explicit transaction. Implicit txns on the control table from SHOW JOBS, are still cool.

Open question: can SHOW JOBS get wrapped in a user's explicit txn.

[dt]

SHOW JOBS, PAUSE JOB, etc are all within the scope of the "jobs system" in my book, as compared to, say, a backup job's individual planning or checkpoint code.

SHOW JOBS can be, but it's a read, so I think that's fine

[msbutler]

per meeting explanation. it could be worth spelling out why the job_status_message table is separated from control_table: this is an attempt to prevent a single row from getting too large (i.e. filling up a single range) and to allow for history of statuses over time.

[msbutler]

clarified in meeting: " user-controlled or job-controlled transactions" implies an explicit transaction. Implicit txns on the control table from SHOW JOBS, are still cool.

Open question: can SHOW JOBS get wrapped in a user's explicit txn.

[petermattis]

I'm not intimately familiar with all of the problems with the singular system.jobs table, but found the rationale behind the separation into multiple tables understandable. I'll leave it to others to give official sign-off on this RFC.

[chengxiong-ruan]

info_id -> info_name / info_description?

[nvanbenschoten]

Reviewed 1 of 1 files at r2, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @chengxiong-ruan, @dt, @miretskiy, @msbutler, @petermattis, and @stevendanna)

---

docs/RFCS/20220608_split_jobs_system_tables.md line 11 at r2 (raw file):

The background job execution framework within CockroachDB currently stores its state and the state
of the jobs it runs in columns of the `system.jobs` table. 

What is the effect of this RFC on the scheduled jobs infrastructure? It's not mentioned, so it doesn't sound like we're proposing any changes to the scheduled_jobs table itself. Do the benefits listed here translate to scheduled jobs execution? I imagine some do. On the other hand, scheduled_jobs.next_run is still mutable, so this may not solve all of the contention issues we've seen during scheduled jobs execution.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 73 at r2 (raw file):

The current system.jobs table will be split into five separate tables: 

1. `system.job_control` tracks the set of jobs, what step they need to take and coordinates which

Is now the time to consider introducing schemas into the system database to organize related tables under a set of high-level functions?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 81 at r2 (raw file):

4. `system.job_intake` is used to sidestep creation transaction contention in the control table.

Each of these four tables serves a specific purpose and is described in more detail below, but two

nit: "but the two"

---

docs/RFCS/20220608_split_jobs_system_tables.md line 229 at r2 (raw file):

system is running, will run or has has failed or creased running, along with human-readable
information about them, such as their description, percentage complete, or  recent status messages;
taken together, these what an operator wants to know when they ask "what jobs is my cluster running?

nit: "these are what"

---

docs/RFCS/20220608_split_jobs_system_tables.md line 235 at r2 (raw file):

reading columns only from the control table and the status tables (plus the intake table). Rendering
the virtual table will _not_ require decoding large proto payloads, that also contain large amounts
of jon-specific state, nor will it block on transactions that are manipulating that state, as the

nit: job-specific

---

docs/RFCS/20220608_split_jobs_system_tables.md line 345 at r2 (raw file):

giant, and such cases should degrade gracefully rather than make the jobs tables unavailable.

Thus, like the status tables above, updates to keys in this info table will instead be an insert of

Implementing MVCC over SQL solely for the purpose of allowing range splits between versions of the row feels unfortunate. That said, you make a strong argument for doing so, especially if this project lands while the MVCC GC TTL on this table is still 25h. How does this project sequence with the line of work to drop the default GC TTL? If we don't want one to depend on the other, could we use a lower default GC TTL specifically on this table, like we do for system.public.replication_stats?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 419 at r2 (raw file):

the only rows skipped by such a SKIP LOCKED query are only new rows

And rows in the process of being consumed from the queue.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 427 at r2 (raw file):

CREATE TABLE system.job_intake (

What are the ordering semantics of this queue? If two jobs are created for the same object, is there any guarantee about which one gets turned into a job first? Are we using mutual exclusion on some other table (e.g. system.descriptors) to prevent incompatible jobs from being queued?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 427 at r2 (raw file):

CREATE TABLE system.job_intake (

Is there observability into the state of this queue? Or some other way to prevent its asynchrony from confusing users? For instance, if a user creates a backup that hasn't made its way out of the jobs_intake table, what will they see when they run SHOW JOBS?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 428 at r2 (raw file):

CREATE TABLE system.job_intake (
id INT PRIMARY KEY,

Is this a sequence? How do we generate new IDs?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 450 at r2 (raw file):

WHERE id IN (
    SELECT * FROM system.job_intake
    FOR UPDATE SKIP LOCKED LIMIT 100;

Without any ordering+predicate here, this is a full table scan over all MVCC garbage in the queue. Are we ok with that because we expect this queue to be very low throughput?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 450 at r2 (raw file):

LIMIT 100

nit: this batch size seems large for a low-throughput queue. My intuition is that batches this large will cause more harm than good, which is what we have seen with scheduled jobs execution. Consider dropping to something like 16.

[dt]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @chengxiong-ruan, @dt, @miretskiy, @msbutler, @nvanbenschoten, @petermattis, and @stevendanna)

---

docs/RFCS/20220608_split_jobs_system_tables.md line 345 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Implementing MVCC over SQL solely for the purpose of allowing range splits between versions of the row feels unfortunate. That said, you make a strong argument for doing so, especially if this project lands while the MVCC GC TTL on this table is still 25h. How does this project sequence with the line of work to drop the default GC TTL? If we don't want one to depend on the other, could we use a lower default GC TTL specifically on this table, like we do for system.public.replication_stats?

Lowering the TTL on this table won't save us since it needs to be backed up, so even if the default TTL is 5min, if the backup cadence is hourly, the protected timestamp will keep it for an hour until that backup runs.

If someone has an 8mb proto they're stuffing into an info record and revise it every 30s, then an hourly backup cadence means we have at least 120 revisions, so almost 2x the range size. Of course it'd be better to not make a single, giant 8mb proto, but I sorta want to be pessimistic here and assume that'll happen one way or another and just defend against it.

[dt]

Scheduling node: only the job_info table is targeting inclusion in 22.2, where it'll be available for opt-in use by the code of existing jobs. The new control, status/progress and intake tables and migrating the job system to them is slated for prototyping (via a standalone simulator app) during the stability period to then be ready for implementation in 23.1.

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @chengxiong-ruan, @dt, @miretskiy, @msbutler, @nvanbenschoten, @petermattis, and @stevendanna)

---

docs/RFCS/20220608_split_jobs_system_tables.md line 179 at r1 (raw file):

Previously, msbutler (Michael Butler) wrote…

nit: info_type is specified in the index

removed

---

docs/RFCS/20220608_split_jobs_system_tables.md line 11 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

What is the effect of this RFC on the scheduled jobs infrastructure? It's not mentioned, so it doesn't sound like we're proposing any changes to the scheduled_jobs table itself. Do the benefits listed here translate to scheduled jobs execution? I imagine some do. On the other hand, scheduled_jobs.next_run is still mutable, so this may not solve all of the contention issues we've seen during scheduled jobs execution.

No touching schedules directly, no. Given the layering boundaries we can just lump schedules in with all other creators of jobs, so the changes to have creator txns not cause job system contention should benefit whether those jobs are created by schedules or humans, but nothing unique to schedules being proposed here.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 73 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Is now the time to consider introducing schemas into the system database to organize related tables under a set of high-level functions?

Eh, I don't know if it really matters; system tables are for use by code in the db, not browsing by humans, so I don't really know if it is worth the added complexity to mess with that just for the extra level of labeling hierarchy?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 321 at r2 (raw file):

Previously, chengxiong-ruan (Chengxiong Ruan) wrote…

info_id -> info_name / info_description?

how about info_key?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 428 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Is this a sequence? How do we generate new IDs?

Callers creating jobs pick them, same as they do job ids today (generally using the unique_rowid logic)

[dt]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @baoalvin1, @chengxiong-ruan, @miretskiy, @msbutler, @nvanbenschoten, @petermattis, and @stevendanna)

---

docs/RFCS/20220608_split_jobs_system_tables.md line 130 at r1 (raw file):

Previously, msbutler (Michael Butler) wrote…

per meeting explanation. it could be worth spelling out why the job_status_message table is separated from control_table: this is an attempt to prevent a single row from getting too large (i.e. filling up a single range) and to allow for history of statuses over time.

Done.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 427 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

What are the ordering semantics of this queue? If two jobs are created for the same object, is there any guarantee about which one gets turned into a job first? Are we using mutual exclusion on some other table (e.g. system.descriptors) to prevent incompatible jobs from being queued?

Yes, external. I've moved this whole intake queue section under a disclaimer that it is speculation and will require its own re-review before it is considered, so we can move forward with the well understood and more urgently needed info table now.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 427 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Is there observability into the state of this queue? Or some other way to prevent its asynchrony from confusing users? For instance, if a user creates a backup that hasn't made its way out of the jobs_intake table, what will they see when they run SHOW JOBS?

I don't think you'd see it in SHOW JOBS until it is in the job table, as SHOW JOBSs should only be jobs that the system actually thinks it is running, i.e. those in the control table.
Maybe a creation statement should poll the queue to see when to return so that creation appears atomic.

We're shelving this whole section to avoid resolving questions like this right now though so we can more forward with the more urgent parts.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 450 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

LIMIT 100

nit: this batch size seems large for a low-throughput queue. My intuition is that batches this large will cause more harm than good, which is what we have seen with scheduled jobs execution. Consider dropping to something like 16.

done.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 450 at r2 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Without any ordering+predicate here, this is a full table scan over all MVCC garbage in the queue. Are we ok with that because we expect this queue to be very low throughput?

I think that's correct about expectation but it deserves considering critically. But re:above, we're not doing this section now.

[knz]

Reviewed 1 of 1 files at r3, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @baoalvin1, @chengxiong-ruan, @dt, @miretskiy, @msbutler, @nvanbenschoten, @petermattis, and @stevendanna)

---

docs/RFCS/20220608_split_jobs_system_tables.md line 72 at r1 (raw file):

Previously, dt (David Taylor) wrote…

we should add first run time here to account for intake lag

did we?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 82 at r1 (raw file):

Previously, dt (David Taylor) wrote…

This column might not be required: I believe ideally we'd let the job itself manage its own retries and any backoff between them it thinks it should add, storing that state in its own info rows, so the backoff at the jobs system level would just be if the node running it crashed before the invoked job-specific function returned, then we should wait some interval before trying again to avoid a job of death crashing all nodes in quick succession.

Could you import the list of columns we ended up implementing into the RFC so it reflects the current situation?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 139 at r1 (raw file):

Previously, dt (David Taylor) wrote…

Fair. I was hoping to punt on many of the implementation nitty gritty, particularly for the new control stuff, initially; our asap priority is to add the new info table, just as an available opt-in tool for existing jobs to use if they want to, with the overall control overhaul being sketched here just to make sure it'll all fit together when we're done

Maybe it's time to have one or two example queries here. As Aditya's experience has shown, these are not trivial.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 190 at r1 (raw file):

Previously, dt (David Taylor) wrote…

We should also limit the max size of a single row.

Did we?

---

docs/RFCS/20220608_split_jobs_system_tables.md line 2 at r3 (raw file):

- Feature Name: Splitting Up the Jobs System Tables
- Status: draft

Change the status.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 5 at r3 (raw file):

- Start Date: 2022-06-08
- Authors: David Taylor, Steven Danna, Yevgeniy Miretskiy
- RFC PR: (PR # after acceptance of initial draft)

Please update these header fields.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 76 at r3 (raw file):

   nodes will do so.
2. `system.job_status_progress` and `system.job_status_message` record human-consumed progress and
   status changes

nit: period at the end of this list item and the following one.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 88 at r3 (raw file):

b) **there may be many rows per job in job_info. **

This change highly severable, allowing it to be implemented gradually: The transition to storing 

This change is*

---

docs/RFCS/20220608_split_jobs_system_tables.md line 132 at r3 (raw file):

CREATE TABLE system.job_control (
	id			INT PRIMARY KEY,

nit: maybe replace tabs by spaces.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 213 at r3 (raw file):

CREATE TABLE system.job_status_progress (

ditto my previous comment about tabs and importing the most recent schema in the code into the RFC.

---

docs/RFCS/20220608_split_jobs_system_tables.md line 326 at r3 (raw file):

CREATE TABLE system.job_info (

ditto: update schema and tabs->spaces

---

docs/RFCS/20220608_split_jobs_system_tables.md line 369 at r3 (raw file):

INSERT INTO system.jobs_info SELECT $1, $3, NOW(), $4 WHERE crdb_internal.sql_liveness_is_alive($2) IS TRUE

Did we end up implemetning this? I think we did the claim below instead. It's worth clarifying in the text.

[dt]

I've slightly adjusted this to reflect the fact we won't be replacing the jobs table but just extending it, and the other tables have new names. I think this still isn't quite is 100% what we expect to implement or indeed already implemented in some places, but I still think it'll be more useful actually in the repo than hanging out on a branch for another two years so I think we should go ahead and merge as is and can amend more later if we're so inclined (though plenty of our RFCs aren't 1:1 with the code by the time it merges anyway).

[dt]

TFTR!

bors r+

[craig]

Build succeeded:

• local_roachtest_fips
• local_roachtest
• lint
• windows_build
• examples_orms
• macos_amd64_build
• acceptance
• macos_arm64_build
• check_generated_code
• docker_image_amd64
• unit_tests
• linux_arm64_build
• linux_amd64_build
• linux_amd64_fips_build