check hangs forever when parallel execution is on

[elenlee]

Hello. There is the following issue with parallel execution.

Suppose there are two files with unused imports:

test_parallel/test1.py

import sys

test_parallel/test2.py

import sys

When run a check executed in parallel (-j 0 by default), it detects the unused imports, but then it hangs forever. And it needs keyboard interrupt to exit:

$ autoflake test_parallel --check

test_parallel/test1.py: Unused imports/variables detected
test_parallel/test2.py: Unused imports/variables detected
^CProcess SpawnPoolWorker-10:
Process SpawnPoolWorker-9:
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-6:
Process SpawnPoolWorker-5:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-1:
Traceback (most recent call last):

Once child process exits after errors found, the main process just sits there waiting for something to be returned from the child process. But child have already exited, so there is nothing to return resulting in a forever wait.

[fsouza]

@giampaolo would you be OK with having a look at this? I can look into it over the long weekend. I'm also planning to augment our test suite in CI to prevent regression.

[giampaolo]

The main process just sits there waiting for something to be returned from the child process. But child have already exited, so there is nothing to return resulting in a forever wait.

Mmm this is weird. Children exiting shouldn't cause parent to hang.
I cannot reproduce this issue. What platform are you on?

[elenlee]

What platform are you on?

I got this on macOS Monterey and also on Centos 7

[giampaolo]

Can you paste the content of all the py files you are linting?

[elenlee]

It can be reproduced with two files from the first post. Each file just needs to have one line of code, i.e. any import never used.

[fsouza]

I can repro with the following steps:

％ mkdir test_parallel
％ echo "import sys" > test_parallel/file1.py
％ cp test_parallel/file{1,2}.py
％ autoflake --check test_parallel/*
test_parallel/file2.py: Unused imports/variables detected
test_parallel/file1.py: Unused imports/variables detected
# hangs

Investigating it now, if I don't succeed I'll update with more info for others.

[fsouza]

Ok, the problem is that we're calling sys.exit and that blocks the pool. #133 fixes it.