Thread watching debounced events hanging #205
Comments
|
Can you use or test the latest 4.x release instead of what's in main? At least to figure out if this is new or existing |
|
I can confirm that the issue also exists in 4.0.12. I tried forcing to 4.0.10, but I do not know why it keeps picking 4.0.12. |
|
I tried changing the delay, but that does not fix anything. Raising the delay to 10s, gives me the highest ever count of processed events (5055), but then it also hangs. |
|
As an aside, just to confirm: when I read your downstream issue you were
going with RawEvents anyway, right? So this is bothersome (and I'd like to
get it fixed!) but not preventing you from your task, if I understand right?
…On Sun, 16 Jun 2019, 08:29 Andy Georges, ***@***.***> wrote:
I tried changing the delay, but that does not fix anything. Raising the
delay to 10s, gives me the highest ever count of processed events (5055),
but then it also hangs.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#205>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABGBC4GJZ6EO6MSAOGHYNTP2VGJ5ANCNFSM4HYGOTGA>
.
|
|
You may need to specify |
|
I'd rather not go with RawEvents for the following reason. In one version of the tool, I'm looking to archive files that reside in a directory. Upon receipt of the CREATE (for the dir), I have no idea in what state the files are in. I tried to recursively watch, but that seemed to miss CREATE events on the files, though I have no immediate idea why. The debounced events will only appear after the delay, if I understood correctly, and then I can be reasonably sure that if the files exist, their contents is also complete. I'll try to pinpoint an earlier version where it still fails later today. |
|
Notify on linux doesn't currently issue creates for files inside
newly-watched directories when doing a recursive descent. I think there's
an issue for that.
…On Sun, 16 Jun 2019, 19:11 Andy Georges, ***@***.***> wrote:
I'd rather not go with RawEvents for the following reason. In one version
of the tool, I'm looking to archive files that reside in a directory. Upon
receipt of the CREATE (for the dir), I have no idea in what state the files
are in. I tried to recursively watch, but that seemed to miss CREATE events
on the files, though I have no immediate idea why. The debounced events
will only appear after the delay, if I understood correctly, and then I can
be reasonably sure that if the files exist, their contents is also
complete. I'll try to pinpoint an earlier version where it still fails
later today.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#205>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABGBC77IETA5AGF7P6EJ4TP2XRSBANCNFSM4HYGOTGA>
.
|
|
Would it issue e.g., CLOSE_WRITE? That would also by ok ;) Or is the problem the underlying inotify implementation in the kernel itself? |
|
If files are created before the watch is placed, they won't emit events. What notify could do is |
|
(That is, the inotify watch depth is one. Given a directory, it will only watch itself and its immediate contents. Recursive watching on Linux is a Notify abstraction on top.) |
|
I'm not sure but I think I've got the same problem, using |
|
I switched to 4.0.13 and get the same behavior. |
We currently can hang up due to notify-rs/notify#205 Signed-off-by: Aron Heinecke <[email protected]>
|
I've also observed this behavior for many months on 4.x.x and finally got to verifying that it only presents itself with the debouncer. I've switched to raw_watcher to get around this. |
|
Version: 4.0.13 I have discovered this same issue on macos (fsevents). It manifests semi-frequently when using the debouncer with a low delay (e.g. 0). I added some logging to confirm the deadlock: ThreadId(18) about to lock: 0x7faea6728190
// ThreadId(18) about to lock: 0x7faea6731380
// ThreadId(18) about to unlock: 0x7faea6731380 (these two cancel out)
ThreadId(3) about to lock: 0x7faea6731380
ThreadId(3) about to lock: 0x7faea6728190
ThreadId(18) about to lock: 0x7faea6731380Diving deeper, the operations buffer lock, and the events lock are held simultaneously, but grabbed in different orders from the two threads. These lines from thread 4 (id 3): And these lines from thread 18 (id 18): |
|
Possibly fixed by #210. |
|
For now, I went with 5.0.0-pre.2, which has no I am fine with that for the moment, and as I understood the debounced part was up for a major rewrite at some point anyway. |
|
Some deadlocks are fixed now by #210 in the 4.X branch. We'll need some input if that resolves the problems and up-port this to 5.X |
|
I encountered exactly same issue on 5.0.0-pre.1. |
…the Graph, and we want to avoid things like notify-rs/notify#205. # Delete this line to force CI to run the JVM tests. [ci skip-jvm-tests]
…the Graph, and we want to avoid things like notify-rs/notify#205. # Delete this line to force CI to run the JVM tests. [ci skip-jvm-tests]
…the Graph, and we want to avoid things like notify-rs/notify#205. # Delete this line to force CI to run the JVM tests. [ci skip-jvm-tests]
### Problem We're on a slightly older version of the `notify` crate because we wanted to use the "debounced" API, which has a thread that attempts to batch events and dedupe them where possible. But the implementation of debouncing has issues like notify-rs/notify#205. ### Solution Move away from the debounced watcher towards the trustier direct delivery of events: post #9636 we do our own debouncing via the Graph not double-cleaning `Nodes`, and via the `--loop` delay. Additionally, fix two cases where code and tests used to race pants' invalidation and assume (intentionally or otherwise) that pants would not notice created files. ### Result Reduced risk of lost `notify` events. [ci skip-jvm-tests]
### Problem We're on a slightly older version of the `notify` crate because we wanted to use the "debounced" API, which has a thread that attempts to batch events and dedupe them where possible. But the implementation of debouncing has issues like notify-rs/notify#205. ### Solution Move away from the debounced watcher towards the trustier direct delivery of events: post #9636 we do our own debouncing via the Graph not double-cleaning `Nodes`, and via the `--loop` delay. Additionally, fix two cases where code and tests used to race pants' invalidation and assume (intentionally or otherwise) that pants would not notice created files. ### Result Reduced risk of lost `notify` events. [ci skip-jvm-tests]
|
Note that since 5.0.0-pre.2 none of the original debouncer code exists any more (and in fact no debouncer for the time being). So this is only v4 related for now. (I'm adding a v4 tag so we can track v5 issues separately.) |
System details
I use notify in https://github.com/itkovian/sarchive/. The issue I opened for this is at itkovian/sarchive#20.
What you did (as detailed as you can)
What you expected
It should just be able to keep track of all events, especially since the queue is big enough.
What happened
The watcher thread and the debouncer (Threads 3 and 2 below) event loop are waiting and seem to be deadlocked. There is no further processing of events.
Relevant code:
The text was updated successfully, but these errors were encountered: