Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve exception parsing #222

Merged
merged 1 commit into from
Jun 14, 2024

Conversation

Delaunay
Copy link
Collaborator

No description provided.

@Delaunay Delaunay merged commit 3577cce into mila-iqia:master Jun 14, 2024
1 of 2 checks passed
@Delaunay Delaunay deleted the better_error_reporting branch June 14, 2024 16:38
@Delaunay
Copy link
Collaborator Author

Example with the newest parsing:

opt-1_3b.local
==============
  * Error code = 1
  * 9 exceptions found
    * 7 x ModuleNotFoundError: No module named 'datasets'
    * 1 x [2024-06-14 11:36:40,300] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1078640) of binary: /milabench/milabench_venv/bin/python
    * 1 x torch.distributed.elastic.multiprocessing.errors.

opt-1_3b.local
==============
  * Error code = 1
  * 9 exceptions found
    * 7 x ModuleNotFoundError: No module named 'datasets'
        | Traceback (most recent call last):
        |   File "/milabench/benchmarks/accelerate_opt/main.py", line 72, in <module>
        |     import datasets
        | ModuleNotFoundError: No module named 'datasets'
    * 1 x [2024-06-14 11:36:40,300] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1078640) of binary: /milabench/milabench_venv/bin/python
        | Traceback (most recent call last):
        |   File "/milabench/benchmarks/accelerate_opt/main.py", line 72, in <module>
        |     import datasets
        | ModuleNotFoundError: No module named 'datasets'
        | [2024-06-14 11:36:40,300] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1078640) of binary: /milabench/milabench_venv/bin/python
    * 1 x torch.distributed.elastic.multiprocessing.errors.
        | Traceback (most recent call last):
        |   File "/milabench/milabench_venv/bin/accelerate", line 8, in <module>
        |     sys.exit(main())
        |              ^^^^^^
        |   File "/milabench/milabench_venv/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
        |     args.func(args)
        |   File "/milabench/milabench_venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1088, in launch_command
        |     multi_gpu_launcher(args)
        |   File "/milabench/milabench_venv/lib/python3.11/site-packages/accelerate/commands/launch.py", line 733, in multi_gpu_launcher
        |     distrib_run.run(args)
        |   File "/milabench/milabench_venv/lib/python3.11/site-packages/torch/distributed/run.py", line 797, in run
        |     elastic_launch(
        |   File "/milabench/milabench_venv/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
        |     return launch_agent(self._config, self._entrypoint, list(args))
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/milabench/milabench_venv/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
        |     raise ChildFailedError(
        | torch.distributed.elastic.multiprocessing.errors.
        | ChildFailedError:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant