Consistency requirements for repo

[travisdowns]

This is more of a question than an issue, please feel free to close if it has been discussed before.

My current solution is to do borg backup locally, and then backup the repository to a remote store.

The way this second step works ensures that individual files are copied atomically, i.e., either the entire new or changed file gets written remotely, or nothing gets written at all - but it does not ensure that the entire set of multiple files modified or created by borg during the last operation gets updated or not. That is, the copy may be partial, but it will always be file-atomic.

So if during a borg create call borg updates 5 files and creates 3 new ones, it is possible that if my remote copy fails part way through, it will only update say 2 of the files and add 1 one of the new ones (later on it will try again and get the missing changes, but if my local host is dead that will never happen).

Will that leave borg in an inconsistent state? Obviously I don't expect to recover the last snapshot whose remote backup was interrupted, but is it possible I can't even recover the earlier ones that were fully backed up and copied to the remote?

If this is unsafe, is there some flag I can add, like an "append only" mode to make it safe?

[ThomasWaldmann]

file-atomic is not enough. "complete or no change" would be best.

In the end, it depends on the order of files being copied (and deleted), but I think this is getting way too much into low-level details.

btw, borg never "updates" segment files, it only writes new ones or deletes old ones.

append-only mode would help a bit as you could better go back to a previous valid state (assuming that only your last copy (copies) was (were) incomplete) using the transaction log.

It might be a bit of an effort though finding a valid state using the log files / return codes of the sync process and looking at the segment files and transaction logs, so I guess I can't recommend that, at least not if it frequently does not complete.

This is also covered in the FAQ and the recommendation is to do multiple borg backups to differently located, independent repositories.

[travisdowns]

btw, borg never "updates" segment files, it only writes new ones or deletes old ones.

The segment files are the numbered files, like 1234?

This is also covered in the FAQ

Thanks, you are referring to this entry, right?

The recommended way of using two repos and borg create against both of them seems better: but it has one major limitation, you need borg running on the remote host, which rules out all the various dumb storage services.

[jdchristensen]

If you are using rsync to copy the repo, you could look at the --delete-delay or --delete-after options, so that file deletions only happen after all of the other updates are done. Also, --delay-updates makes the updating closer to being atomic. You might even try a two pass system:

• rsync the data subdirectory, without deletions, so new segment files are copied over.
• rsync the whole repo with --delete-delay. This copies the index over and then deletes old segment files.

Then old backups should be accessible even if either step gets interrupted. (I think. Untested.) You might need to run borg check --repair ...

[jdchristensen]

Another way to do an atomic rsync is to do a hard link copy of the remote repo into repo2 (very fast, uses little space), rsync the local repo to the remote repo2, and when that successfully terminates, delete the old remote repo.

[DiagonalArg]

I'm just switching from restic to borg, because I trust borg's encryption better. With restic, I was having the same issue regarding remote backups, and so using rsync to the remote. I found that using --delay-updates --delete-delay --partial-dir and --temp-dir, you can get something that's almost atomic, but with some corner cases that weren't clear and which might depend on restic's behavior. So I wrote my own atomic version. Comments welcome:

#!/bin/bash

HOME="/home/foo"

LOCALFOLDER="$HOME/Backup/Borg"
LOCALSUBFOLDER="safe"
LOCALSOURCE="$LOCALFOLDER/$LOCALSUBFOLDER"
LOCALLOG="$LOCALSOURCE.log"

REMOTEHOST="[email protected]"
REMOTEHOME="/home/foo"
REMOTEFOLDER="$REMOTEHOME/Backup"
REMOTESUBFOLDER="safe"
REMOTETARGET="$REMOTEFOLDER/$REMOTESUBFOLDER"

NICERSYNC="nice -n 19 ionice -c 3 /usr/bin/rsync"
SWITCHES="-acvi --partial-dir=\"$REMOTETARGET-partial\""

errormsgandexit() {
    if [[ $1 -ne 0 ]]; then
	echo "$2"
	exit $1
    fi
}

ssherrormsgandexit() {
    SSHCONNECTERROR=255
    if [[ $1 -eq $SSHCONNECTERROR ]]; then
	echo "$2"
	exit $1
    fi
}
    
echo -e "\n$(date) Rsync $LOCALSUBFOLDER \n--------------------" >> "$LOCALLOG"

# If TARGET exists, we use that for links:
ssh $REMOTEHOST "[[ -d \"$REMOTETARGET\" ]]" &>> "$LOCALLOG" && \
    SWITCHLINKS="--link-dest=\"$REMOTETARGET\""
ssherrormsgandexit $? "ssh connect failure while checking $REMOTETARGET directory." >> "$LOCALLOG"

# Find the smallest n such that TARGET.n doesn't exist. use TARGET.1-(n-1) for links.
declare -i n=1
while ssh $REMOTEHOST "[[ -d \"$REMOTETARGET.$n\" ]]" &>> "$LOCALLOG"; ! ((RETURN=$?)); do
    SWITCHLINKS="$SWITCHLINKS --link-dest=\"$REMOTETARGET.$((n++))\""
done
ssherrormsgandexit $RETURN "ssh connect failure while checking $REMOTETARGET.$n directory." >> "$LOCALLOG"

SWITCHESLINKS="$SWITCHES $SWITCHLINKS"

# now I am free to copy to TARGET.n:
$NICERSYNC $SWITCHESLINKS "$LOCALSOURCE/" $REMOTEHOST:\"$REMOTETARGET.$n\" &>> "$LOCALLOG"
errormsgandexit $? "rsync error, code $?." >> "$LOCALLOG"

# if rsync was successful, just keep the most recent complete TARGET
ssh $REMOTEHOST "([[ ! -d \"$REMOTETARGET\" ]] || rm -r \"$REMOTETARGET\") && \
        	mv \"$REMOTETARGET.$n\" \"$REMOTETARGET\" && \
    		([[ ! -d \"$REMOTETARGET.1\" ]] || rm -r \"$REMOTETARGET.\"*)" &>> "$LOCALLOG"
RETURN=$?
ssherrormsgandexit $RETURN "ssh connect failure while reordering $REMOTETARGET\* directories." >> "$LOCALLOG"
errormsgandexit $RETURN "error when reorganizing $REMOTETARGET* directories." >> "$LOCALLOG"

[ThomasWaldmann]

OK, now I am curious: what led you to trusting borg's encryption more than restic's?

I don't know these rsync options, but if the result is a 100% copy of a stable, inactive state of a borg repo, it will be fine as a repo backup.

Just don't use 2 borg repos with same repo ID with borg, that can lead to crypto breakage as well as to other unwanted effects.

borg < 2.0 has some known limitations / potential issues due to its usage of AES-CTR mode. borg tries its best to make pretty sure to never reuse nonce values, but there can be problematic situations if the counter value memory is lost on the client and the repo is in a not trustworthy or older state or if multiple clients use the same repo. This is known and documented.

In borg2 (current master branch, not finished yet), AES-CTR will be removed and replaced by crypto that does not need this kind of nonce/counter managment. See the internals docs of master branch for more details.

[DiagonalArg]

@ThomasWaldmann

This is all we have regarding restic's crypto. An hour in the airport doesn't sound like much of a code review to me!

As for borg, I don't intend to use the same repo from more than one client, and AES seems straightforward enough. The fact that the docs have some discussion of the security issues, gives me some element of confidence. (I remember reading more around the web, but I'm not quite sure what. I did see this, but the upshot didn't appear to come to much.) I didn't find anything like that for restic. And, in a different area, I found them impervious to discussion.

If there's more I should know, please do tell!

[ThomasWaldmann]

Well, FOSS projects often have no funds to pay a professionally done security review, so they get more or less for-free review.

Guess, the borg2 crypto is much better (and I am currently not aware of any issues), but more review for that would be great.