c2c: write a protected timestamp on the destination cluster #92093
Assignees
Labels
A-disaster-recovery
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-disaster-recovery
Comments
adityamaru
added
C-enhancement
|
cc @cockroachdb/disaster-recovery |
adityamaru
added a commit
to adityamaru/cockroach
that referenced
this issue
Nov 22, 2022
During C2C replication as the destination tenant is ingesting KVs we must protect a certain window of MVCC revisions from garbage collection so that the user can cutover to any of the timestamps that lie within this window. To this effect we introduce a `ReplicationTTLSeconds` field to the replication job payload that governs the size of this window relative to the replication job's highwatermark (frontier timestamp). On the first resumption of the replication job we write a protected timestamp record on the destination tenant's keyspace protecting all revisions above `now()`. As the replication job updates its highwatermark, the PTS record is pulled up to protect above `highWatermark - ReplicationTTLSeconds`. This active management of the PTS always ensures that users can cutover to any time in (highWatermark-ReplicationTTLSeconds, highWatermark] and older revisions are gradually made eligible for GC as the frontier progresses. The PTS is released if the replication job fails or is cancelled. Fixes: cockroachdb#92093 Release note: None
craig bot
pushed a commit
that referenced
this issue
Nov 23, 2022
92336: streamingest: write and manage PTS on the destination tenant r=stevendanna a=adityamaru During C2C replication as the destination tenant is ingesting KVs we must protect a certain window of MVCC revisions from garbage collection so that the user can cutover to any of the timestamps that lie within this window. To this effect we introduce a `ReplicationTTLSeconds` field to the replication job payload that governs the size of this window relative to the replication job's highwatermark (frontier timestamp). On the first resumption of the replication job we write a protected timestamp record on the destination tenant's keyspace protecting all revisions above `now()`. As the replication job updates its highwatermark, the PTS record is pulled up to protect above `highWatermark - ReplicationTTLSeconds`. This active management of the PTS always ensures that users can cutover to any time in (highWatermark-ReplicationTTLSeconds, highWatermark] and older revisions are gradually made eligible for GC as the frontier progresses. The PTS is released if the replication job fails or is cancelled. Fixes: #92093 Release note: None Co-authored-by: adityamaru <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Data is streamed from the source to the destination tenant over several partitions. A frontier timestamp that is updated during ingestion, tracks the highest timestamp up to which all partitions on the destination cluster have completed streaming. As time progresses and more data is ingested, versions that fall below the destination tenant's GCThreshold will become eligible for GC. Timestamps below the tenant's GCThreshold become ineligible for cutover since those version are no longer available to revert to.
To ensure that the user always has some generous window of timestamps to cutover to, we must write a protected timestamp over the ingesting tenant's keyspan that protects at
FrontierTimestamp - C2CTTL, where C2CTTL can default to 25hours to begin with. This can later be made a configurable value that is specified when the replication stream is being setup. As the frontier moves forward, this protected timestamp must be pulled up as well so as to make data outside the C2CTTL window eligible for GC.Jira issue: CRDB-21570
Epic: CRDB-18749
The text was updated successfully, but these errors were encountered: