Invalid cross-device link: backup fails when a symlinked folder is replaced by its real destination folder on source

[jean-christophe-manciot]

Ubuntu 17.10
bit 1.1.24

Configuration:

@Germar

• Backup on 12/19 succeeds with the following symlinked folder which is not backed up (cf. configuration):

# ll
Bash -> /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/Bash/

• Backup on 12/21 fails when "Bash" symlink is replaced by the real folder and all its subfolders which have been copied from destination /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/Bash/ to source:

# ll
Bash/

BIT log:

[E] Error: rsync: link "/media/actionmystique/WD-Backup-Luks/MSI-GE60-Backup/backintime/SAMSUNG-Ubuntu/root/1/new_snapshot/backup/media/actionmystique/MSI-GE60-Ubuntu/home/actionmystique/Bash/CLI Commands.png" => ../../20171219-170725-263/backup/media/actionmystique/MSI-GE60-Ubuntu/home/actionmystique/Bash/CLI Commands.png failed: Invalid cross-device link (18)
...
[E] Error: rsync: link "/media/actionmystique/WD-Backup-Luks/MSI-GE60-Backup/backintime/SAMSUNG-Ubuntu/root/1/new_snapshot/backup/media/actionmystique/MSI-GE60-Ubuntu/home/actionmystique/Bash/Scripts/yang2tree-module.sh" => ../../20171219-170725-263/backup/media/actionmystique/MSI-GE60-Ubuntu/home/actionmystique/Bash/Scripts/yang2tree-module.sh failed: Invalid cross-device link (18)

My understanding is that bit tries to symlink the current backed up files inside Bash with the same files already backed up 2 days ago, although nothing was actually backed up because their sources were at that time only ... inside a symlinked folder.

On 12/21, bit should have backed up the real folder this time, even though nothing had changed when compared to 2 days ago, because there was no real previous backup.

Let le know if you need some clarification.

[buhtz]

@jean-christophe-manciot There is a new forming maintaining team and we do review all issues. Is this problem still relevant for you or did you find a solution?

Tag: Feedback

I think we should write some FAQ content about how to handle symlinks.

[jean-christophe-manciot]

This issue has not been solved with bit 1.3.2.

[buhtz]

Thanks for the feedback. Can you please run BIT from terminal in debug mode via backintime-qt --debug and post the complete output here.

[jean-christophe-manciot]

starting bit:

backintime-qt --debug
DEBUG: [common/backintime.py:583 argParse] Arguments: {'debug': True} | unknownArgs: []

Back In Time
Version: 1.3.2

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

DEBUG: [common/backintime.py:670 getConfig] config file: /root/.config/backintime/config
DEBUG: [common/backintime.py:671 getConfig] share path: /root/.local/share/backintime
DEBUG: [common/backintime.py:672 getConfig] profiles: 1=Main profile, 8=<profile_name>
DEBUG: [common/pluginmanager.py:90 PluginManager.load] Register plugin path /usr/share/backintime/plugins
DEBUG: [common/pluginmanager.py:106 PluginManager.load] Add plugin notifyplugin.py
DEBUG: [common/pluginmanager.py:106 PluginManager.load] Add plugin usercallbackplugin.py
DEBUG: [common/pluginmanager.py:106 PluginManager.load] Add plugin qt4plugin.py
DEBUG: [plugins/usercallbackplugin.py:46 UserCallbackPlugin.callback] Call user-callback: /root/.config/backintime/user-callback 1 Main profile 5
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
DEBUG: [common/tools.py:805 keyringSupported] No keyring due to import error.
DEBUG: [common/mount.py:73 Mount.__init__] pw-cache is not running
DEBUG: [common/mount.py:81 Mount.__init__] Call command: /usr/bin/backintime pw-cache start
DEBUG: [plugins/usercallbackplugin.py:46 UserCallbackPlugin.callback] Call user-callback: /root/.config/backintime/user-callback 1 Main profile 7
DEBUG: [common/tools.py:1237 readCrontab] Read 4 lines from users crontab
DEBUG: [common/config.py:1466 Config.removeOldCrontab] Clearing system Back In Time entries
DEBUG: [common/config.py:1500 Config.cronLine] Profile: Main profile | Automatic backup: Every Day
DEBUG: [common/config.py:1500 Config.cronLine] Profile: <profile_name> | Automatic backup: Disabled
DEBUG: [common/tools.py:1274 writeCrontab] Wrote 4 lines to users crontab

exiting bit

DEBUG: [common/tools.py:805 keyringSupported] No keyring due to import error.
DEBUG: [common/mount.py:73 Mount.__init__] pw-cache is not running
DEBUG: [common/mount.py:81 Mount.__init__] Call command: /usr/bin/backintime pw-cache start
DEBUG: [plugins/usercallbackplugin.py:46 UserCallbackPlugin.callback] Call user-callback: /root/.config/backintime/user-callback 1 Main profile 8
DEBUG: [plugins/usercallbackplugin.py:46 UserCallbackPlugin.callback] Call user-callback: /root/.config/backintime/user-callback 1 Main profile 6

[AlanDeSmet]

I've been running into this with Back In Time 1.3.2 from the official(?) PPA. I'm on Ubuntu 22.04.1 LTS.

The relevant files look like this (names changed mostly for simplicity):

• $HOME — The only entry in Include
• "NoBackup" — entry in my Exclude list
• $HOME/NoBackup/ — Directory full of large files I don't need backed up
• $HOME/NoBackup/LargePackage/ — Directory of specific files in question
• $HOME/src/MyProject/LargePackage — symbolic link to $HOME/NoBackup/LargePackage/ ; the link is an absolute path. So it "works" in the backup, as it's pointing to the original.

All worked as expected, then $HOME/src/MyProject/LargePackage changed from a symlink to an actual directory full of files. Now it fails with many error messages in the log that look like

[E] Error: rsync: [generator] link "/media/adesmet/adesmet-backup/nimbostratus-backintime/backintime/nimbostratus/adesmet/1/new_snapshot/backup/home/adesmet/src/cspp-geo-aitf/lib/ShellB3/README.txt" => ../../20221103-090001-365/backup/home/adesmet/src/cspp-geo-aitf/lib/ShellB3/README.txt failed: Invalid cross-device link (18)
[E] Error: rsync: [generator] link "/media/adesmet/adesmet-backup/nimbostratus-backintime/backintime/nimbostratus/adesmet/1/new_snapshot/backup/home/adesmet/src/cspp-geo-aitf/lib/ShellB3/VERSION.txt" => ../../20221103-090001-365/backup/home/adesmet/src/cspp-geo-aitf/lib/ShellB3/VERSION.txt failed: Invalid cross-device link (18)
[E] Error: rsync: [generator] link "/media/adesmet/adesmet-backup/nimbostratus-backintime/backintime/nimbostratus/adesmet/1/new_snapshot/backup/home/adesmet/src/cspp-geo-aitf/lib/ShellB3/requires" => ../../20221103-090001-365/backup/home/adesmet/src/cspp-geo-aitf/lib/ShellB3/requires failed: Invalid cross-device link (18)

My notifications get similarly filled; I believe they're the same error messages.

I've been trying without success to create a small reproducible case. I'll try some more, but can't spend much more time on it.

If there are specific things you'd like to try or data that I could collect that might help, please let me know!

[aryoda]

Invalid cross-device link (18)

I could not find this error msg in BiT directly and the output [E] Error: rsync: [generator] link... indicates that BiT does log a caught error of rsync (but I can't see where exactly it happens in the Code).

Is anyone able to produce a minimal reproducible example to make this problem "debugable"?

[jean-christophe-manciot]

@aryoda
Everything there is to know to "produce a minimal reproducible example" is already explained in the first post.

[aryoda]

@jean-christophe-manciot OK, THX for reporting and clarifying this. We need another developer to step in here who has experiences with this none-trival scenario (EncFS + symbolic links + "copy links" enabled) to safe our rare resources.

Currently we are preparing a stabilized release of BiT (mainly bug fixes of HIGH bugs) so please give us some time to come back once we find time for this.

[AlanDeSmet]

I believe I have a very small, if not minimal reproducible example for what I'm seeing. It seems similar to what the original reporter is seeing.

This is with Back In Time 1.3.2 from the official(?) PPA. I'm on Ubuntu 22.04.1 LTS. My system rsync identifies itself as rsync version 3.2.3 protocol version 31.

The key series of events appears to be:

1. Make a backup containing a symlink to a directory outside of the backup.
2. Replace the symlink with a copy of the directory. At this point, the files in the directory must be older than the symlink was. If the files are newer, the next backup completes without complaint.
3. Make a few backup: observe "Invalid cross-device" error

Attached should be a script that reproducibly does that for me. It's a bit wonky since the backup is in the background; it'll pause waiting for you to hit Enter when the first back up finishes. It relies on and will clobber /tmp/work, /tmp/NoBackup, and /dev/shm/backintime/x/x/1; paths can be adjusted early in the script.

repro.txt

[jean-christophe-manciot]

I'm baffled by the fact that this issue is dismissed and considered as so unimportant that it will probably never be addressed (very far away, someday, until "never").

Actually, it is quite a serious one because every time a symlink pointing to a folder is replaced by a regular folder, backintime fails to take the snapshot and no backup is done at all!

[buhtz]

Dear Jean,
I understand your point.

This issue is not dismissed, only closed issues are. Please read the milestone carefully and look at its bright side: It also means it could be fixed until the uber-next release (1.3.6 or 1.4.0) which is quit near.
It is all about resources and priorities. Please look into the other issues, the code base and the test suite. To our current view there are much more urgent problems. But I also agree that this issue is serious.

If you want to help with deeper analysis of the relevant code and an approach for a solution, I would be glad to assist you. If you provide code that don't need to be perfect. Just throw something in and we figuring out together.

Have you considered that this might be an rsync-based issue? Maybe this is the usual rsync-behaviour?

[jean-christophe-manciot]

@buhtz

AFAIK, there is nothing wrong with rsync, although some of its options are sometimes complex.
I suspect that you use the option --keep-dirlinks although you shouldn't:

       --keep-dirlinks, -K
              This option causes the receiving side to treat a symlink to a directory  as  though
              it  were a real directory, but only if it matches a real directory from the sender.
              Without this option, the receiver's symlink would be deleted and  replaced  with  a
              real directory.

... the last sentence being exactly what we want.

POC:
Let's suppose that we have in /tmp:

test -> /home/actionmystique/Downloads/tmp/test/
test_dest_folder/

Trying to rsync the test folder symlink from /tmp into /tmp/test_dest_folder:

rsync --acls --archive --group --human-readable --hard-links --info=progress2 --numeric-ids --owner --perms --specials --stats --times --xattrs test test_dest_folder
...
ll test_dest_folder
total 76
drwxr-xr-x  2 root root  4096 Jul  3 11:57 ./
drwxrwxrwt 25 root root 69632 Jul  3 11:53 ../
lrwxrwxrwx  1 root root    39 Jul  3 11:53 test -> /home/actionmystique/Downloads/tmp/test/

No surprise.
Now, if we replace the source /tmp/test symlink by its referent inside /tmp:

ll test*
test:
total 80
drwxr-xr-x  3 root root  4096 Jul  3 11:43 ./
drwxrwxrwt 26 root root 69632 Jul  3 12:00 ../
drwxr-xr-x  2 root root  4096 Jul  3 11:43 inner_test/

test_dest_folder:
total 76
drwxr-xr-x  2 root root  4096 Jul  3 11:57 ./
drwxrwxrwt 26 root root 69632 Jul  3 12:00 ../
lrwxrwxrwx  1 root root    39 Jul  3 11:53 test -> /home/actionmystique/Downloads/tmp/test/

Trying again to rsync the test real folder from /tmp into /tmp/test_dest_folder with the same CLI command:

rsync --acls --archive --group --human-readable --hard-links --info=progress2 --numeric-ids --owner --perms --specials --stats --times --xattrs test test_dest_folder
...
ll test_dest_folder
total 80
drwxr-xr-x  3 root root  4096 Jul  3 12:03 ./
drwxrwxrwt 26 root root 69632 Jul  3 12:00 ../
drwxr-xr-x  3 root root  4096 Jul  3 11:43 test/

As you can see, not using the --keep-dirlinks allowed us to replace the test symlink by the real test folder inside the destination folder.

FYI, here are the options used in BIT:

[buhtz]

Thanks for analyzing this. I can't find that switch in our rsync call but it might be inherited by one of the others switches.

[aryoda]

I could reproduce the problem (while working on #1491) using the script of @AlanDeSmet
(excellent script to make the issue reproducible, THX a lot for that!)

The actual rsync cmd (with paths replaced by ...) is:

rsync --recursive --times --devices --specials --hard-links --human-readable -s --links --perms --executability --group --owner --info=progress2 --no-inc-recursive --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20230808-234435-315/backup --chmod=Du+wx --exclude=... --exclude=... --exclude=... --exclude=* / .../backintime/x/x/1/new_snapshot/backup

The relevant rsync error messages and the exit code are:

rsync: [generator] link "/.../test/subdir/file" => .../test/subdir/file failed: Invalid cross-device link (18)
[...]
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]
~/temp  > echo $?
23

The exit code 23 means "Partial transfer due to error" according to man rsync.

Observations:

• The rsync cmd generated by BiT does not contain --keep-dirlinks
• The problem occurs only if the the backup source folder ("rsync include") is on a different device (mount) than the backup destination (as noted in the repro.txt script of @AlanDeSmet; see lines 5 - 7)
• the error occurs when the linked folder is replaced by a copy of the folder itself (with the same timestamps using cp -a).
  I did not test what happens when the timestamp is different.

Next steps:

• Find an rsync specialist who can advice us on how to solve this kind of problem (preferably by using rsync options that do always work)
• Fix the rsync call in BiT
• Optionally: Write an FAQ entry on how to cope with this special cross-device symlink constellation