Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing RunitimeError for Windows #118

Merged
merged 3 commits into from
Jun 22, 2020

Conversation

fjribi
Copy link
Contributor

@fjribi fjribi commented Jun 19, 2020

PR concerning errors like ("NameError: name 'np' is not defined") when using multiprocess exec mode on windows:
Linked to #104 & #93

Is it convenient to add the following "workaround code" to \cadCAD_init_.py file to fixe the issue?

name = "cadCAD"
configs = []

#added code
import os
if os.name == 'nt': 
    import dill
    dill.settings['recurse'] = True 
This fixed the issue.

or should we only add the code to \cadCAD\engine_init_.py] file to fixe the issue?

class ExecutionContext:
    def __init__(self, context: str = ExecutionMode.multi_proc) -> None:
...
        elif context == 'multi_proc':
           #add code here? 
           import os
           if os.name == 'nt': 
               import dill
               dill.settings['recurse'] = True 
            self.method = parallelize_simulations

to limit the setting to multiprocess exec mode and reduce potential side effects. Indeed, although this seems to be a stable workaround for windows, there are some comments (see [https://github.com/uqfoundation/dill/issues/115]) that are mitigated about the solution.

There are 2 other less convincing alternatives for our issue:
1 - To declare the missing global variables (which are not correctly serialized without dill.settings['recurse'] = True) at the function level (see [https://github.com/uqfoundation/multiprocess/issues/6]) => could be less practical, since this requires to re-declare global variables locally...
2 - to use cloudpickle instead of dill (see [https://github.com/apache/airflow/issues/7870]), => no idea on the impact of the migration

Regards,

JEJodesty and others added 3 commits June 15, 2020 16:42
Concerning errors like ("NameError: name 'np' is not defined") when using multiprocess exec mode on windows: Linked to cadCAD-org#104 & cadCAD-org#93
@JEJodesty JEJodesty requested review from tylerdmace and JEJodesty and removed request for tylerdmace June 20, 2020 17:51
@JEJodesty JEJodesty added enhancement New feature or request and removed enhancement New feature or request labels Jun 20, 2020
@JEJodesty JEJodesty changed the title Multi-processes Exec Mode multiprocessing RunitimeError for WIndows Jun 22, 2020
@JEJodesty JEJodesty changed the title multiprocessing RunitimeError for WIndows multiprocessing RunitimeError for Windows Jun 22, 2020
@JEJodesty
Copy link
Member

JEJodesty commented Jun 22, 2020

@fjribi
Thanks for your contribution!
Replacing location of your change to cadCAD/__init__.py post merge

@JEJodesty JEJodesty merged commit a48c123 into cadCAD-org:staging Jun 22, 2020
@fjribi
Copy link
Contributor Author

fjribi commented Jun 22, 2020

Ok, thanks

JEJodesty added a commit that referenced this pull request Nov 19, 2020
multiprocessing RunitimeError for Windows
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants