Unison (fastcheck=true) doesn't notice new files (written in exFAT by another computer running Windows) #1122
Comments
Artur2048
added
the
defect
|
Are you using the CLI without Could you use ktrace (ktruss, strace, not sure what Linux calls it) to see what system calls unison executes? Basically the key question is if the operating system is returning information about these files. Or perhaps -- guessing wildly -- it has cached information and does not. Have you looked at the filesystem back on the Linux machine to see that the files you think are there are there? |
|
yes. i'm using CLI without -repeat, One thing that puzzles me is - if i create additional file on my local Linux machine and run Unison the second time - it WILL actually detect the new file i just created as well as the file that was added on another PC. I think it is possible Unison gets cached information... but it's not cached by OS... it's maybe cached in those ar* and fp* files in my .unison directory Also (not sure if it matters) - time on both machines is synchronized via NTP. Sure.. I'll try to run strace and post the results. |
|
I have a suspicion of what is going on here... but first, could you check a few things:
|
|
And then, please post the output of
|
|
OK. i will retest with (the problem is - the issue does not happen every single time....) But I have reproduced it again and gathered the information you were asking for content of the profile file: Replicas are now in sync. New folder and new file is on USB: But as for now - it's not on the local disk: let's compare amount of files (and subdirectories) in the Hi-Res folder: Starting Unison to resync running the strace shows that Unison is calling lstat: Stat of the directory (and the subdirectory) before adding new file now i'll add NEW file on my local Linux machine:
Stat of the directory after adding file: And let's start Unison again As we can see - after adding another file on the LOCAL machine - Unison can now see the changes including other - previously undetected file. (but only in the Hi-Res folder) |
|
Thank you for all the detailed data. You didn't produce quite exactly what I asked for (just stat the parent dir before and after adding a new file on the other machine, don't add anything on the local machine) but I believe I got my suspicion confirmed based on the existing data anyway. Now, this is not 100% confirmed yet but I believe what is happening here is:
We should now confirm that this is indeed what is happening and figure out why the directory modification time is not being updated. |
|
I have retested with fastcheck=false here's 2 replicas with different content: (the Extreme Pro has New Folder and New Text Document.txt added on another machine) let's try first WITHOUT adding fastcheck=false: Then i ran it again with added fastcheck false: now i chose "Q" to terminate reconciliation process and get the stats of directories before and after adding file... This time i'll provide stats on all 3 levels Stats below Now i'll add another file on local machine to the "CD Quality" folder: and let's see stats again: let's rerun unison (without adding fastcheck=false) Hmm... it's quite unusual. I've read the description of fastcheck... i would assume it affects the way files are analyzed (not directories). For example if a file has unchanged modifiication date and size - Unison will assume that the file was not changed. Please let me know if you would like to get any more information or retests. |
|
Ah. my bad. I guess you wanted to see stats of the "CD Quality" directory BEFORE i added file on the other computer... |
|
OK. reproduced it again, and you were right!
ok - before unmounting (at this point replicas are really in sync): now disk was unmounted and file was added on Windows PC and the stats indeed are not changed - modification times are the same as before: And after either - adding some file in "Hi-Res" folder or running Unison with -fastcheck false - it synchronizes directories properly. So.. the other machine where i was adding files (that didn't cause modification times to update) is Windows 11 PC. File system on my USB-SSD drive is ExFat |
|
Maybe we could have Unison crawl directories in the regular way ("find ./") and don't rely on their modification times? crawling directory does not take that much time... it would add a fraction of second maybe to the total time Not sure how Windows treats directories on ExFat removable drives and why the modification times are not updated properly... (I confirm i always unmount my drives safely on Linux as well as on Windows) |
|
Very good, at least we now know what the underlying problem is.
Right. The directories part was bolted on to it later, as an "experimental" feature 15 years ago ( Lines 538 to 539 in 77cd088
Yes, disabling fastcheck is definitely not an option. As to whether always crawling directories is a performance penalty or not, I have no clue. It apparently could have been in 2009 when this optimization was added. Before deciding whether to disable or remove this optimization, it would be great to have some real world testing results on various filesystems and media (I'm specifically thinking of high-latency media like rotating drives and USB sticks). I will first prepare binaries suitable for testing. Regardless of the corrective action we decide to take, it would still be interesting to find out why the modification time is not updated. exFAT specification seems to require that modification time be updated https://learn.microsoft.com/en-us/windows/win32/fileio/exfat-specification#746-lastmodifiedtimestamp-lastmodified10msincrement-and-lastmodifiedutcoffset-fields This random page I found seems to suggest that the modification time should be updated when adding a file: https://answers.microsoft.com/en-us/windows/forum/all/exfat-format-external-drive-with-all-folders/39d33729-2148-4718-84f5-419336a49f81 Edit: found an interesting commit 05faade |
|
My quick reactions are
And:
if this situation is really that Windows doesn't update the mod time, please file a bug with MS, and then provide a link to the public bugtracker entry. (I have no idea how that goes, but in theory paid-for proprietary software comes with support.) But first make sure you are running a version of Windows that is currently receiving support and fully up to date. |
gdt
changed the title
|
I've retitled now that we understand more, and marked this as a Windows issue (happy to remove that if we find that the not-updating-mtime behavior is more widespread). |
gdt
added
discuss
This needs to be measured. Do we know if there really is any meaningful cost and impact on caching? The previously synced individual files need to be checked anyway. This could be tested separately with a tiny C program perhaps. Alt A: read the dir contents and stat all files in the dir. Alt B: don't read the dir contents and stat all the files in the dir. Of course, to execute alt B, the list of files needs to be provided externally, without reading the dir contents.
Yes it does, this was demonstrated above.
According to the commit from 2009 that I referenced, unlikely. |
Interestingly enough, there appears to be a significant difference. I added some code to enable/disable the optimization in question by an env var and ran some primitive tests. A local sync between two replicas of 400 000 files, no subdirs, no updates ("Nothing to do: replicas have not changed since last sync."). Cache warm, not purged at any time during testing. While the runtime increase is 30-40%, it is less than one second for 800 000 files. It would be interesting to see the effect of a cold cache and high-latency media. Edit: testing with 10 000 files per replica (which should be more representative of an average user) also shows overall time increased by 0.01-0.02 seconds (this increase also correlates well with the one above). This is more significant than I'd imagined. Edit2: trying to clear caches, with both replicas above the overall execution time increases (hopefully indicating that caches indeed were purged) but the delta between dirFastCheck=true vs false remained the same. |
|
For those interested in doing some testing of their own, download a binary from https://github.com/tleedjarv/unison/actions/runs/13329824560 With that binary you can use an environment variable Set the env var to |
|
And harder to test, but the effect of cache loading on other workloads is also important. If somebody is running this once, that's one thing, but if it's -repeat, cache-changing behavior is more serious. Reading the 2nd link, it sounds like someone was finding that changing a file didn't update the directory mod time, and it was explained that this is correct, but if you add a file, it's ok. (The answer is a little hard to understand, because it talks about how things are designed vs the specification.) Regarding the 2009 commit, good find! But that seems to be not really the right conditional. With the benefit of lots of hindsight, it seems the right condition is "filesystem is FAT and was written by Windows". Which you can't tell. Having |
Is there a meaningful difference (actual cache changing) between the two alternatives I outlined in #1122 (comment) ? I'm not challenging your statement, I'm just trying to wrap my head around it. So far, I'm thinking that the observed performance difference can be attributed in part to OCaml's Unix library and Unison's own code doing more work (which could have optimization potential) and in part to a modern Linux kernel having an inefficient implementation of But this is more a point of curiosity now. As for the actual fix to the issue at hand my current thinking is this:
|
|
What I meant is that reading all the dirs is going to load data into some kind of fs caches in the OS, and displace other data. That is causing to cause other programs to see different timing behavior. That's really hard to figure out, but not dismissable. I do see your point that It could be that on all systems that stat calls on the files force the same loading. I don't think it's clear there is no need for a preference because it's been 15 years. (Really windows is buggy and people running windows should file a bug and MS should fix it; there's no need to change unison, which is not wrong. And therefore nothing to do at all.) Another view is that this usage is really unusual, if it hasn't been reported yet, and affected people disabling fastcheck is an ok workaround. But seriously, we are talking about accommodating buggy proprietary software, to what extent, and how. Let's wait to see how the bug report gets made and handled at least for a bit. People may need to disable fastcheck on dirs but not files for FAT written by WIndows, if "support" of proprietary software is as illusory as seems reasonably likely. As I understand it, the problem is not FAT; it's Windows mis-handling FAT, and so the disabling should be for FAT which has been written by Windows. I don't see how that can be automatic. I also don't see how we can detect the underlying fs in any reasonable or reliable way. There could be remote fileysystems, union or null mounts, whatever. I'm also not inclined to disable something which is sound according to standards, in a way that's hidden or harms non-broken usage. |
|
Disabling Disabling the dir optimization on the other hand is very much an option. Disabling it has no known downsides. (Also note that it was never enabled on Windows; we could now enable it for NTFS and ReFS, for example) We can of course do one better and not disable it universally. If we disable the dir optimization when having detected FAT then we've covered a vast majority (and I'm thinking 99.9+% here) of potential issue cases. It doesn't matter that we don't detect FAT behind layers. It doesn't matter that we don't know if Windows has or has not touched it. Not only because these would be a very tiny minority of the cases. More importantly because, again, disabling the optimization has no downsides. I dislike the idea of exposing another obscure preference that nobody knows how or when to use. Even when having the preference, the heavy lifting should still be done by auto-detection, leaving manual tuning for expert users and debugging purposes. The fact that this issue has seemingly not been reported before is very strange to me. I would have thought that people syncing with USB sticks or drives with (ex)FAT filesystems was a slightly common sight. Since this hasn't come up before, I can only assume that people syncing to FAT drives are either setting |
|
This issue seems to be 100% about working around a Windows bug. Assuming that's true (and contrary evidence of course welcome), ethically it's on MS and proprietary software users to fix that. It is fundamentally unreasonable to expect volunteer Free Software developers to accomodate buggy proprietary software because people somehow are ok with that software not being fixed. I would like to defer everything about this until the OP files a bug with MS and we see the resolution. |
|
Unfortunately Microsoft does not care about things like that... and would probably not bother to fix it. As i understand the simple and effective "workaround" is already present in Windows version of Unison. With option Please keep in mind that even if you magically make Microsoft fix their Win-11 system, there are other systems which will remain unpatched (Win-10, Win-7, etc..) Last, but not least - ExFAT system is used in MANY devices and their SD cards:
Unison could be used to synchronize contents of SD card which was written by Nikon camera, or Samsung tablet, or any other device... Also, if someone is using ExFAT on a Linux machine - there's probably a reason. |
|
Sounds like you are decliing to file a bug report. |
|
If I may chime in... Expecting any change from Microsoft or any other proprietary software vendor here is purely wishful thinking. It just won't happen and we all know that. Even if some vendor fixed their bug, a large portion of users would still be stuck with a version that just doesn't get that update, because the vendor dictates who gets the update, and simple business economics tells me that they will limit the target audience to the absolutely necessary minimum. Personal experience shows this pattern even with smaller companies, in my case Synology. After several attempts and even some friendly discussions with real persons calling themselves "product management", it all just didn't lead anywhere. Some FOSS-head requesting access to an existing, half-documented API is just economically uninteresting. So insisting on ethical responsibility doesn't help anybody. Free Software exists because people had a motivation to improve their situation and let others benefit from that work for free. What would the FOSS landscape look like if we had just considered Microsoft Windows' lacking POSIX compliance a "bug on their end"? Time spent on such a discussion is mostly wasted IMHO. It's quite reasonable to add a workaround for a specific system's bugs if that system happens to be used by a large portion of the user base. Especially if Unison's resulting behavior may silently cause troubles that go unnoticed, depending on some very specific file access pattern (added only, no other changes in the same dir, using a sensible OS on the file system). |
Instructions
I have thoroughly read the Reporting Bugs and Feature Requests wiki page.
Meta
Running Unison to sync local drive with removable USB SSD.
Environment
Reproduction Recipe
After performing initial sync with Unison, I've unmounted and removed my USB SSD, plugged it to another PC and added some files and directories.
I have then plugged it back to my computer and started Unison again. After rescanning both folders it wrote that "Everything is in sync" (which is not true - there are new files there)
Interesting fact - if I create additional file in exactly the same directory as the "undetected new files" on my local computer and rescan with Unison - it will detect all new files - including files created on another PC which were previously undetected
Additional information - my SSD is formatted as ExFAT, and the profile PRF is:
Steps to reproduce:
for example add file "test1" in already existing directory "Archive"
and also add "test2" in the already existing directory "Documents"
Expected behavior vs actual behavior
Expected behavior - Unison should be able to detect files modified to removable drives on external PCs
Additionally:
The text was updated successfully, but these errors were encountered: