Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Only trim logs to min persisted lsn across all known nodes #1781

Closed
Tracked by #1675
tillrohrmann opened this issue Aug 2, 2024 · 0 comments · Fixed by #1783
Closed
Tracked by #1675

Only trim logs to min persisted lsn across all known nodes #1781

tillrohrmann opened this issue Aug 2, 2024 · 0 comments · Fixed by #1783
Assignees
@tillrohrmann

Comments

@tillrohrmann
Copy link
Contributor

tillrohrmann commented Aug 2, 2024 •
edited
Loading

Until we have support for creating a state snapshot and making this state snapshot accessible to all nodes, we must not trim the log if one of the known nodes lags behind. If we wanted to support that new nodes can join the cluster at any point in time, then we must never trim the log because a new node will have to replay the log from the beginning for a given partition processor.
@tillrohrmann tillrohrmann mentioned this issue Aug 2, 2024
Closed
@tillrohrmann tillrohrmann self-assigned this Aug 2, 2024
tillrohrmann added a commit to tillrohrmann/restate that referenced this issue Aug 2, 2024
@tillrohrmann
Check if all nodes have reported a persisted lsn before trimming the log
c0b7f97 
Until we can share partition processor snapshots between Restate nodes (e.g.
by fetching them from S3), we can only trim the log if all known nodes have
reached the trim point. Otherwise, we risk that a node that is currently not
available needs log entries which were trimmed. One crucial assumption is
that no new nodes will join the cluster once the first log trimming has
happened. For this to work, we also need the sharing of partition processor
snapshots.

This fixes restatedev#1781.
tillrohrmann added a commit to tillrohrmann/restate that referenced this issue Aug 2, 2024
@tillrohrmann
Check if all nodes have reported a persisted lsn before trimming the log
76e3023 
Until we can share partition processor snapshots between Restate nodes (e.g.
by fetching them from S3), we can only trim the log if all known nodes have
reached the trim point. Otherwise, we risk that a node that is currently not
available needs log entries which were trimmed. One crucial assumption is
that no new nodes will join the cluster once the first log trimming has
happened. For this to work, we also need the sharing of partition processor
snapshots.

This fixes restatedev#1781.
@tillrohrmann tillrohrmann mentioned this issue Aug 2, 2024
Merged
tillrohrmann added a commit to tillrohrmann/restate that referenced this issue Aug 6, 2024
@tillrohrmann
Check if all nodes have reported a persisted lsn before trimming the log
0485712 
Until we can share partition processor snapshots between Restate nodes (e.g.
by fetching them from S3), we can only trim the log if all known nodes have
reached the trim point. Otherwise, we risk that a node that is currently not
available needs log entries which were trimmed. One crucial assumption is
that no new nodes will join the cluster once the first log trimming has
happened. For this to work, we also need the sharing of partition processor
snapshots.

This fixes restatedev#1781.
@tillrohrmann tillrohrmann mentioned this issue Aug 8, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tillrohrmann tillrohrmann

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Check if all nodes have reported a persisted lsn before trimming the log tillrohrmann/restate
1 participant
@tillrohrmann