Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock with futures mpsc and rayon_wait #396

Closed
kaksmet opened this issue Jul 8, 2017 · 9 comments
Closed

Deadlock with futures mpsc and rayon_wait #396

kaksmet opened this issue Jul 8, 2017 · 9 comments
Labels
bug

Comments

@kaksmet
Copy link

kaksmet commented Jul 8, 2017

I'm spawning some tasks using rayon, where each task is partially dependent on the previous task. futures::sync::mpsc together with rayon::spawn_future seemed like a good solution to manage this dependency but I'm having problems with deadlocks.

Running the code provided below with two threads in the rayon threadpool works as expected

$ RAYON_NUM_THREADS=2 RUSTFLAGS='--cfg rayon_unstable' cargo run
4 blocked
3 blocked
2 blocked
1 blocked
0 sent
1 unblocked
1 sent
2 unblocked
2 sent
3 unblocked
3 sent
4 unblocked
4 sent

but with three threads it deadlocks quite reliably

$ RAYON_NUM_THREADS=3 RUSTFLAGS='--cfg rayon_unstable' cargo run
1 blocked
4 blocked
2 blocked
3 blocked
0 sent

extern crate futures;
extern crate rayon;

use futures::{Sink, Stream};
use futures::sync::mpsc as futures_mpsc;
use std::{thread, time};

fn main() {
    let size = 5;

    rayon::scope(|scope| {
        let mut prev_receiver = None;

        for i in 0..size {
            let (sender, receiver) = futures_mpsc::channel(1);

            scope.spawn(move |_| {
                do_work(i, prev_receiver.take(), sender);
            });

            prev_receiver = Some(receiver);
        }
    });
}

fn do_work(
    number: usize,
    mut receiver: Option<futures_mpsc::Receiver<()>>,
    mut sender: futures_mpsc::Sender<()>,
) {
    if let Some(receiver) = receiver.take() {
        let receiver_future = receiver.into_future();
        println!("{} blocked", number);
        let _ = rayon::spawn_future(receiver_future).rayon_wait();
        println!("{} unblocked", number);
    }

    thread::sleep(time::Duration::from_millis(10));

    let _ = sender.start_send(());
    println!("{} sent", number);
}
@cuviper
Copy link
Member

cuviper commented Oct 17, 2017

Have you tried this lately? It works fine for me on the current master rayon-futures, at least.

@kaksmet
Copy link
Author

kaksmet commented Oct 17, 2017

Still deadlocks on master for me with a 4-core machine running macOS 10.12.6

Here's the version I ran, slightly modified to compile with rayon-futures from git:

Cargo.toml

[package]
name = "korv-test"
version = "0.1.0"
authors = ["Ulf Nilsson <[email protected]>"]

[dependencies]
rayon = { git = "https://github.com/rayon-rs/rayon" }
rayon-futures = { git = "https://github.com/rayon-rs/rayon" }
futures = "0.1.16"

main.rs

extern crate futures;
extern crate rayon;
extern crate rayon_futures;

use futures::{Sink, Stream};
use futures::sync::mpsc as futures_mpsc;
use rayon_futures::ScopeFutureExt;
use std::{thread, time};

fn main() {
    let size = 5;

    rayon::scope(|scope| {
        let mut prev_receiver = None;

        for i in 0..size {
            let (sender, receiver) = futures_mpsc::channel(1);

            scope.spawn(move |_| {
                do_work(i, prev_receiver.take(), sender);
            });

            prev_receiver = Some(receiver);
        }
    });
}

fn do_work(
    number: usize,
    mut receiver: Option<futures_mpsc::Receiver<()>>,
    mut sender: futures_mpsc::Sender<()>,
) {
    if let Some(receiver) = receiver.take() {
        let receiver_future = receiver.into_future();
        println!("{} blocked", number);
        //let _ = rayon::spawn_future(receiver_future).rayon_wait();
        let _ = rayon::ThreadPool::global().spawn_future(receiver_future).rayon_wait();
        println!("{} unblocked", number);
    }

    thread::sleep(time::Duration::from_millis(10));

    let _ = sender.start_send(());
    println!("{} sent", number);
}

Here's the output from running with RAYON_LOG enabled

$ RAYON_NUM_THREADS=3 RAYON_LOG=1 RUSTFLAGS='--cfg rayon_unstable' cargo run
    Updating git repository `https://github.com/rayon-rs/rayon`
    Updating registry `https://github.com/rust-lang/crates.io-index`
   Compiling rayon-core v1.2.2 (https://github.com/rayon-rs/rayon#2aa274c6)
   Compiling futures v0.1.16
   Compiling lazy_static v0.2.9
   Compiling scopeguard v0.3.3
   Compiling either v1.3.0
   Compiling libc v0.2.32
   Compiling coco v0.1.1
   Compiling rand v0.3.17
   Compiling num_cpus v1.7.0
   Compiling rayon v0.8.2 (https://github.com/rayon-rs/rayon#2aa274c6)
   Compiling rayon-futures v0.1.0 (https://github.com/rayon-rs/rayon#2aa274c6)
   Compiling korv-test v0.1.0 (file:///Users/ulf/korv-test)
    Finished dev [unoptimized + debuginfo] target(s) in 5.98 secs
     Running `target/debug/korv-test`
InjectJobs { count: 1 }
WaitUntil { worker: 1 }
WaitUntil { worker: 2 }
WaitUntil { worker: 0 }
UninjectedWork { worker: 1 }
DidNotFindWork { worker: 0, yields: 0 }
DidNotFindWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 0, yields: 1 }
DidNotFindWork { worker: 2, yields: 1 }
DidNotFindWork { worker: 2, yields: 2 }
DidNotFindWork { worker: 0, yields: 2 }
DidNotFindWork { worker: 2, yields: 3 }
DidNotFindWork { worker: 0, yields: 3 }
StoleWork { worker: 0, victim: 1 }
StoleWork { worker: 2, victim: 1 }
FoundWork { worker: 0, yields: 4 }
JobCompletedOk { owner_thread: 1 }
FoundWork { worker: 2, yields: 4 }
WaitUntil { worker: 1 }
1 blocked
FoundWork { worker: 1, yields: 0 }
4 blocked
WaitUntil { worker: 1 }
WaitUntil { worker: 2 }
FoundWork { worker: 1, yields: 0 }
FoundWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
3 blocked
StoleWork { worker: 2, victim: 1 }
WaitUntil { worker: 1 }
FoundWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
2 blocked
DidNotFindWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 1, yields: 1 }
WaitUntil { worker: 2 }
FoundWork { worker: 2, yields: 0 }
DidNotFindWork { worker: 1, yields: 2 }
DidNotFindWork { worker: 2, yields: 0 }
DidNotFindWork { worker: 1, yields: 3 }
DidNotFindWork { worker: 2, yields: 1 }
DidNotFindWork { worker: 1, yields: 4 }
DidNotFindWork { worker: 2, yields: 2 }
DidNotFindWork { worker: 1, yields: 5 }
DidNotFindWork { worker: 2, yields: 3 }
DidNotFindWork { worker: 1, yields: 6 }
DidNotFindWork { worker: 1, yields: 7 }
DidNotFindWork { worker: 2, yields: 4 }
DidNotFindWork { worker: 1, yields: 8 }
DidNotFindWork { worker: 2, yields: 5 }
DidNotFindWork { worker: 1, yields: 9 }
DidNotFindWork { worker: 2, yields: 6 }
DidNotFindWork { worker: 2, yields: 7 }
DidNotFindWork { worker: 2, yields: 8 }
DidNotFindWork { worker: 1, yields: 10 }
DidNotFindWork { worker: 2, yields: 9 }
DidNotFindWork { worker: 2, yields: 10 }
DidNotFindWork { worker: 1, yields: 11 }
DidNotFindWork { worker: 2, yields: 11 }
DidNotFindWork { worker: 1, yields: 12 }
DidNotFindWork { worker: 2, yields: 12 }
DidNotFindWork { worker: 2, yields: 13 }
DidNotFindWork { worker: 1, yields: 13 }
DidNotFindWork { worker: 2, yields: 14 }
DidNotFindWork { worker: 1, yields: 14 }
DidNotFindWork { worker: 2, yields: 15 }
DidNotFindWork { worker: 1, yields: 15 }
DidNotFindWork { worker: 1, yields: 16 }
DidNotFindWork { worker: 2, yields: 16 }
DidNotFindWork { worker: 1, yields: 17 }
DidNotFindWork { worker: 2, yields: 17 }
DidNotFindWork { worker: 1, yields: 18 }
DidNotFindWork { worker: 1, yields: 19 }
DidNotFindWork { worker: 2, yields: 18 }
DidNotFindWork { worker: 1, yields: 20 }
DidNotFindWork { worker: 2, yields: 19 }
DidNotFindWork { worker: 1, yields: 21 }
DidNotFindWork { worker: 1, yields: 22 }
DidNotFindWork { worker: 1, yields: 23 }
DidNotFindWork { worker: 2, yields: 20 }
DidNotFindWork { worker: 1, yields: 24 }
DidNotFindWork { worker: 1, yields: 25 }
DidNotFindWork { worker: 2, yields: 21 }
DidNotFindWork { worker: 1, yields: 26 }
DidNotFindWork { worker: 2, yields: 22 }
DidNotFindWork { worker: 1, yields: 27 }
DidNotFindWork { worker: 1, yields: 28 }
DidNotFindWork { worker: 2, yields: 23 }
DidNotFindWork { worker: 1, yields: 29 }
DidNotFindWork { worker: 2, yields: 24 }
DidNotFindWork { worker: 1, yields: 30 }
DidNotFindWork { worker: 2, yields: 25 }
DidNotFindWork { worker: 1, yields: 31 }
DidNotFindWork { worker: 2, yields: 26 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 27 }
GetSleepy { worker: 1, state: 0 }
DidNotFindWork { worker: 2, yields: 28 }
GotSleepy { worker: 1, old_state: 0, new_state: 4 }
DidNotFindWork { worker: 1, yields: 33 }
DidNotFindWork { worker: 1, yields: 34 }
DidNotFindWork { worker: 2, yields: 29 }
DidNotFindWork { worker: 1, yields: 35 }
DidNotFindWork { worker: 1, yields: 36 }
DidNotFindWork { worker: 2, yields: 30 }
DidNotFindWork { worker: 1, yields: 37 }
DidNotFindWork { worker: 2, yields: 31 }
DidNotFindWork { worker: 1, yields: 38 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 39 }
DidNotFindWork { worker: 1, yields: 40 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 41 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 42 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 43 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 44 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 45 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 46 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 47 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 48 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 49 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 50 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 51 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 52 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 53 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 54 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 55 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 56 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 57 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 58 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 59 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 60 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 61 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 62 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 4 }
DidNotFindWork { worker: 1, yields: 63 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 64 }
GetSleepy { worker: 2, state: 4 }
FellAsleep { worker: 1 }
DidNotFindWork { worker: 2, yields: 32 }
GetSleepy { worker: 2, state: 1 }
GotSleepy { worker: 2, old_state: 1, new_state: 7 }
DidNotFindWork { worker: 2, yields: 33 }
DidNotFindWork { worker: 2, yields: 34 }
DidNotFindWork { worker: 2, yields: 35 }
DidNotFindWork { worker: 2, yields: 36 }
DidNotFindWork { worker: 2, yields: 37 }
DidNotFindWork { worker: 2, yields: 38 }
DidNotFindWork { worker: 2, yields: 39 }
DidNotFindWork { worker: 2, yields: 40 }
DidNotFindWork { worker: 2, yields: 41 }
DidNotFindWork { worker: 2, yields: 42 }
DidNotFindWork { worker: 2, yields: 43 }
DidNotFindWork { worker: 2, yields: 44 }
DidNotFindWork { worker: 2, yields: 45 }
DidNotFindWork { worker: 2, yields: 46 }
DidNotFindWork { worker: 2, yields: 47 }
DidNotFindWork { worker: 2, yields: 48 }
DidNotFindWork { worker: 2, yields: 49 }
DidNotFindWork { worker: 2, yields: 50 }
DidNotFindWork { worker: 2, yields: 51 }
DidNotFindWork { worker: 2, yields: 52 }
DidNotFindWork { worker: 2, yields: 53 }
DidNotFindWork { worker: 2, yields: 54 }
DidNotFindWork { worker: 2, yields: 55 }
DidNotFindWork { worker: 2, yields: 56 }
DidNotFindWork { worker: 2, yields: 57 }
DidNotFindWork { worker: 2, yields: 58 }
DidNotFindWork { worker: 2, yields: 59 }
DidNotFindWork { worker: 2, yields: 60 }
DidNotFindWork { worker: 2, yields: 61 }
DidNotFindWork { worker: 2, yields: 62 }
DidNotFindWork { worker: 2, yields: 63 }
DidNotFindWork { worker: 2, yields: 64 }
FellAsleep { worker: 2 }
Tickle { worker: 0, old_state: 1 }
0 sent
GotAwoken { worker: 1 }
GotAwoken { worker: 2 }
JobCompletedOk { owner_thread: 1 }
StoleWork { worker: 1, victim: 0 }
DidNotFindWork { worker: 2, yields: 0 }
FoundWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 0, yields: 0 }
DidNotFindWork { worker: 2, yields: 1 }
DidNotFindWork { worker: 1, yields: 0 }
DidNotFindWork { worker: 0, yields: 1 }
DidNotFindWork { worker: 2, yields: 2 }
DidNotFindWork { worker: 1, yields: 1 }
DidNotFindWork { worker: 0, yields: 2 }
DidNotFindWork { worker: 2, yields: 3 }
DidNotFindWork { worker: 1, yields: 2 }
DidNotFindWork { worker: 0, yields: 3 }
DidNotFindWork { worker: 2, yields: 4 }
DidNotFindWork { worker: 1, yields: 3 }
DidNotFindWork { worker: 0, yields: 4 }
DidNotFindWork { worker: 2, yields: 5 }
DidNotFindWork { worker: 1, yields: 4 }
DidNotFindWork { worker: 0, yields: 5 }
DidNotFindWork { worker: 2, yields: 6 }
DidNotFindWork { worker: 1, yields: 5 }
DidNotFindWork { worker: 0, yields: 6 }
DidNotFindWork { worker: 2, yields: 7 }
DidNotFindWork { worker: 1, yields: 6 }
DidNotFindWork { worker: 0, yields: 7 }
DidNotFindWork { worker: 2, yields: 8 }
DidNotFindWork { worker: 1, yields: 7 }
DidNotFindWork { worker: 0, yields: 8 }
DidNotFindWork { worker: 2, yields: 9 }
DidNotFindWork { worker: 1, yields: 8 }
DidNotFindWork { worker: 0, yields: 9 }
DidNotFindWork { worker: 2, yields: 10 }
DidNotFindWork { worker: 1, yields: 9 }
DidNotFindWork { worker: 0, yields: 10 }
DidNotFindWork { worker: 2, yields: 11 }
DidNotFindWork { worker: 0, yields: 11 }
DidNotFindWork { worker: 1, yields: 10 }
DidNotFindWork { worker: 2, yields: 12 }
DidNotFindWork { worker: 0, yields: 12 }
DidNotFindWork { worker: 2, yields: 13 }
DidNotFindWork { worker: 1, yields: 11 }
DidNotFindWork { worker: 0, yields: 13 }
DidNotFindWork { worker: 2, yields: 14 }
DidNotFindWork { worker: 1, yields: 12 }
DidNotFindWork { worker: 0, yields: 14 }
DidNotFindWork { worker: 2, yields: 15 }
DidNotFindWork { worker: 1, yields: 13 }
DidNotFindWork { worker: 0, yields: 15 }
DidNotFindWork { worker: 2, yields: 16 }
DidNotFindWork { worker: 0, yields: 16 }
DidNotFindWork { worker: 1, yields: 14 }
DidNotFindWork { worker: 2, yields: 17 }
DidNotFindWork { worker: 0, yields: 17 }
DidNotFindWork { worker: 1, yields: 15 }
DidNotFindWork { worker: 2, yields: 18 }
DidNotFindWork { worker: 1, yields: 16 }
DidNotFindWork { worker: 0, yields: 18 }
DidNotFindWork { worker: 2, yields: 19 }
DidNotFindWork { worker: 1, yields: 17 }
DidNotFindWork { worker: 0, yields: 19 }
DidNotFindWork { worker: 1, yields: 18 }
DidNotFindWork { worker: 2, yields: 20 }
DidNotFindWork { worker: 0, yields: 20 }
DidNotFindWork { worker: 1, yields: 19 }
DidNotFindWork { worker: 2, yields: 21 }
DidNotFindWork { worker: 0, yields: 21 }
DidNotFindWork { worker: 1, yields: 20 }
DidNotFindWork { worker: 2, yields: 22 }
DidNotFindWork { worker: 1, yields: 21 }
DidNotFindWork { worker: 0, yields: 22 }
DidNotFindWork { worker: 2, yields: 23 }
DidNotFindWork { worker: 1, yields: 22 }
DidNotFindWork { worker: 0, yields: 23 }
DidNotFindWork { worker: 2, yields: 24 }
DidNotFindWork { worker: 0, yields: 24 }
DidNotFindWork { worker: 1, yields: 23 }
DidNotFindWork { worker: 2, yields: 25 }
DidNotFindWork { worker: 0, yields: 25 }
DidNotFindWork { worker: 1, yields: 24 }
DidNotFindWork { worker: 2, yields: 26 }
DidNotFindWork { worker: 0, yields: 26 }
DidNotFindWork { worker: 1, yields: 25 }
DidNotFindWork { worker: 2, yields: 27 }
DidNotFindWork { worker: 0, yields: 27 }
DidNotFindWork { worker: 1, yields: 26 }
DidNotFindWork { worker: 2, yields: 28 }
DidNotFindWork { worker: 0, yields: 28 }
DidNotFindWork { worker: 1, yields: 27 }
DidNotFindWork { worker: 0, yields: 29 }
DidNotFindWork { worker: 2, yields: 29 }
DidNotFindWork { worker: 1, yields: 28 }
DidNotFindWork { worker: 2, yields: 30 }
DidNotFindWork { worker: 0, yields: 30 }
DidNotFindWork { worker: 1, yields: 29 }
DidNotFindWork { worker: 2, yields: 31 }
DidNotFindWork { worker: 1, yields: 30 }
DidNotFindWork { worker: 0, yields: 31 }
DidNotFindWork { worker: 2, yields: 32 }
DidNotFindWork { worker: 1, yields: 31 }
GetSleepy { worker: 2, state: 0 }
DidNotFindWork { worker: 0, yields: 32 }
GotSleepy { worker: 2, old_state: 0, new_state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 33 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 34 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 35 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 36 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 37 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 38 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 39 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 40 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 41 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 42 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 43 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 44 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 45 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 46 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 47 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 48 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 49 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 50 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 51 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 52 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 53 }
GetSleepy { worker: 1, state: 6 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 54 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 55 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 56 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 57 }
DidNotFindWork { worker: 0, yields: 32 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 58 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 59 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 60 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 2, yields: 61 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 0, state: 6 }
DidNotFindWork { worker: 2, yields: 62 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 2, yields: 63 }
GetSleepy { worker: 0, state: 6 }
GetSleepy { worker: 1, state: 6 }
DidNotFindWork { worker: 2, yields: 64 }
DidNotFindWork { worker: 0, yields: 32 }
DidNotFindWork { worker: 1, yields: 32 }
FellAsleep { worker: 2 }
GetSleepy { worker: 0, state: 1 }
GetSleepy { worker: 1, state: 1 }
GotSleepy { worker: 0, old_state: 1, new_state: 3 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 33 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 34 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 35 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 36 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 37 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 38 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 39 }
DidNotFindWork { worker: 0, yields: 40 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 41 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 42 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 43 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 44 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 45 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 46 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 47 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 48 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 49 }
DidNotFindWork { worker: 0, yields: 50 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 51 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 52 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 53 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 54 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 55 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 56 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 57 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 58 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 59 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 60 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 61 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 0, yields: 62 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 63 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 3 }
DidNotFindWork { worker: 1, yields: 32 }
DidNotFindWork { worker: 0, yields: 64 }
GetSleepy { worker: 1, state: 3 }
FellAsleep { worker: 0 }
DidNotFindWork { worker: 1, yields: 32 }
GetSleepy { worker: 1, state: 1 }
GotSleepy { worker: 1, old_state: 1, new_state: 5 }
DidNotFindWork { worker: 1, yields: 33 }
DidNotFindWork { worker: 1, yields: 34 }
DidNotFindWork { worker: 1, yields: 35 }
DidNotFindWork { worker: 1, yields: 36 }
DidNotFindWork { worker: 1, yields: 37 }
DidNotFindWork { worker: 1, yields: 38 }
DidNotFindWork { worker: 1, yields: 39 }
DidNotFindWork { worker: 1, yields: 40 }
DidNotFindWork { worker: 1, yields: 41 }
DidNotFindWork { worker: 1, yields: 42 }
DidNotFindWork { worker: 1, yields: 43 }
DidNotFindWork { worker: 1, yields: 44 }
DidNotFindWork { worker: 1, yields: 45 }
DidNotFindWork { worker: 1, yields: 46 }
DidNotFindWork { worker: 1, yields: 47 }
DidNotFindWork { worker: 1, yields: 48 }
DidNotFindWork { worker: 1, yields: 49 }
DidNotFindWork { worker: 1, yields: 50 }
DidNotFindWork { worker: 1, yields: 51 }
DidNotFindWork { worker: 1, yields: 52 }
DidNotFindWork { worker: 1, yields: 53 }
DidNotFindWork { worker: 1, yields: 54 }
DidNotFindWork { worker: 1, yields: 55 }
DidNotFindWork { worker: 1, yields: 56 }
DidNotFindWork { worker: 1, yields: 57 }
DidNotFindWork { worker: 1, yields: 58 }
DidNotFindWork { worker: 1, yields: 59 }
DidNotFindWork { worker: 1, yields: 60 }
DidNotFindWork { worker: 1, yields: 61 }
DidNotFindWork { worker: 1, yields: 62 }
DidNotFindWork { worker: 1, yields: 63 }
DidNotFindWork { worker: 1, yields: 64 }
FellAsleep { worker: 1 }
^C

@cuviper
Copy link
Member

cuviper commented Oct 26, 2017

Hmm, OK, this reproduces for me on Linux, but not every time. 1 or 2 threads are consistently fine, 3-7 threads will hang most times (but not always!), then 8 threads seems OK again.

@cuviper
Copy link
Member

cuviper commented Oct 26, 2017

FWIW, instead of throwing away the inner scope handle in scope.spawn(move |_| { ..., you could pass that scope to do_work to call scope.spawn_future. But it doesn't change the deadlock behavior for me.

@cuviper
Copy link
Member

cuviper commented Oct 26, 2017

This seems to be a dependency inversion. e.g. with future B depending on future A via mpsc. Suppose a given thread executes do_work() -> A.rayon_wait() and it's not ready yet. This job is the one that will send() to B. While A is blocked, it steals some other job, which then executes do_work() -> B.rayon_wait(). Now we're blocked probing for B, no longer checking A. Then something completes A, but we don't notice, and we never get to make progress.

@cuviper
Copy link
Member

cuviper commented Oct 26, 2017

Reduced example:

extern crate futures;
extern crate rayon;
extern crate rayon_futures;

use futures::{Sink, Stream};
use futures::sync::mpsc as futures_mpsc;
use rayon_futures::ScopeFutureExt;

fn main() {
    let config = rayon::Configuration::new().num_threads(1);
    let pool = rayon::ThreadPool::new(config).unwrap();
    pool.scope(|scope| {
        let (mut sender1, receiver1) = futures_mpsc::channel(1);
        let (mut sender2, receiver2) = futures_mpsc::channel(1);

        scope.spawn(move |_| {
            sender1.start_send(()).unwrap();
        });

        scope.spawn(move |scope| {
            scope.spawn_future(receiver2.into_future()).rayon_wait().unwrap();
        });

        scope.spawn(move |scope| {
            scope.spawn_future(receiver1.into_future()).rayon_wait().unwrap();
            sender2.start_send(()).unwrap();
        });
    });
}

With a single thread, the spawns will run LIFO:

  • wait on channel 1, look for more work
    • wait on channel 2, look for more work
      • send to channel 1
    • still waiting for channel 2...

With more threads, the order can be arbitrarily mixed, since local threads steal LIFO and remote threads steal FIFO. So fixing this isn't just a matter of spawning in a better order.

@cuviper cuviper added the bug label Oct 26, 2017
@nikomatsakis
Copy link
Member

Hmm, interesting. Yes, I think the idea of rayon_wait was sort of flawed. We should probably just remove it. My original intention with the futures work was certainly not to encourage people to write using rayon_wait -- that was supposed to be a "fallback".

I originally thought we would solve these sorts of things by growing additional threads, but I suppose that this example shows that even that isn't always a solution. (I can't tell you how many times I have rediscovered this exact deadlock pattern -- that is, one caused by the implicit edge of using the same stack -- in my life.)

@cuviper
Copy link
Member

cuviper commented Dec 15, 2017

FWIW, Future::wait was deprecated in rust-lang/futures-rs#634. I guess they now have Blocking::wait, but I don't know if we need the same -- or even if we can, given the bug here.

@cuviper cuviper mentioned this issue Feb 5, 2018
Open
@cuviper
Copy link
Member

cuviper commented Feb 24, 2023

This code was removed in #716.

@cuviper cuviper closed this as not planned Won't fix, can't repro, duplicate, stale Feb 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cuviper @nikomatsakis @kaksmet