Add ClearMLHandler to track all MONAI Experiments

[skinan]

ClearML is a leading MLOps stack that can supercharge dialogues research with its state-of-the-art experiment tracking capability. ClearML: https://clear.ml/

I have added a clearml_handler.py which contains ClearMLHandler, ClearMLStatsHandler, and ClearMLImageHandler class. Basically, clearml can track everything which is tracked by tensorboard including scalars, and debug samples, and can also store models, artifacts, and the console in the ClearML server which can be easily accessed from ClearML WebUI shown as below:

ClearMLStatsHandler and ClearMLImageHandler can be used with Pytorch Trainer just like TensorboardStatsHandler and TensorboardImageHandler.

Use ClearMLStatsHandler() & ClearMLImageHandler(log_dir="./runs/", batch_transform=from_engine(["image", "label"]),output_transform=from_engine(["pred"]),) with any MONAI example to test its functionality.

Also, please let us where should we put further documentation and tutorials regarding this MLOps tool in the MONAI.

[Nic-Ma]

Thanks for the contribution.
Please add some unit tests to cover the feature.

Thanks.

[skinan]

Working on adding some unittests.

[skinan]

@wyli , @Nic-Ma , @binliunls, is the memory ran out of error on these Tests normal? The failing tests are blocking the PR.

[wyli]

@wyli , @Nic-Ma , @binliunls, is the memory ran out of error on these Tests normal? The failing tests are blocking the PR.

yes, that issue is probably introduced by the code changes in this PR. any idea what the root cause is? we can skip the windows test using @skip_if_windows

MONAI/tests/utils.py

Line 247 in b85e2c6

def skip_if_windows(obj):

but preferably we should understand the issue first.

[skinan]

@wyli , Could you please clarify what the coverage command is doing here: https://github.com/Project-MONAI/MONAI/actions/runs/4352352552/jobs/7605037274#step:12:18505 . Because, we need to understand why it has triggered the clearml fire binding. As it was not supposed to trigger clearml.

Also, I would request that can you please check monitoring on the runner machine, so we can understand if it truly ran out of memory.

[wyli]

@wyli , Could you please clarify what the coverage command is doing here: https://github.com/Project-MONAI/MONAI/actions/runs/4352352552/jobs/7605037274#step:12:18505 . Because, we need to understand why it has triggered the clearml fire binding. As it was not supposed to trigger clearml.

Also, I would request that can you please check monitoring on the runner machine, so we can understand if it truly ran out of memory.

that's testing the bundle command:

python -m monai.bundle run --runner_id evaluating ...

you can run the test locally with:

python -m tests.test_integration_bundle_run

@Nic-Ma @binliunls could you please help the debugging here?

[Nic-Ma]

/build

[wyli]

looking at the error messages, it seems the error is from the clearml fire binding on windows.

     component_trace = _Fire(component, args, parsed_flag_args, context, name)\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\fire\\core.py", line 475, in _Fire\r
    component, remaining_args = _CallAndUpdateTrace(\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\frameworks\\__init__.py", line 36, in _inner_patch\r
    raise ex\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\frameworks\\__init__.py", line 34, in _inner_patch\r
    ret = patched_fn(original_fn, *args, **kwargs)\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\fire_bind.py", line 172, in __CallAndUpdateTrace\r
    PatchFire.__groups, PatchFire.__commands = PatchFire.__get_all_groups_and_commands(\r
  File "C:\\hostedtoolcache\\windows\\Python\\3.8.10\\x64\\lib\\site-packages\\clearml\\binding\\fire_bind.py", line 249, in __get_all_groups_and_commands\r
    PatchFire._commands_sep.join(query_group) + PatchFire._commands_sep if len(query_group) > 0 else ""\r

https://github.com/Project-MONAI/MONAI/actions/runs/4371805968/jobs/7648041295

[wyli]

/build

[wyli]

the docker test is blocked by the unit test https://github.com/Project-MONAI/MONAI/actions/runs/4397243843/jobs/7700566567 could you please help debug it? @skinan

the test is running on a github runner https://github.com/Project-MONAI/MONAI/actions/runs/4397243843/workflow#L83-L99

[skinan]

@wyli , This is weird. We are trying to reproduce it locally, to find out what goes wrong.

[wyli]

@wyli , This is weird. We are trying to reproduce it locally, to find out what goes wrong.

thanks, I try to reproduce it on github by manually triggering the test dev...wyli:MONAI:trigger-tests but it works fine now https://github.com/wyli/MONAI/actions/runs/4415391874 I'll monitor this test for the dev branch