Error while Training from scratch

[vincentochs]

Hi,

(sorry in advance for stupid questions but I am kind of new to this field).

When I am trying to label my data from scratch following this tutorial:

https://www.youtube.com/watch?v=3HTh2dqZqew

I get the following error:

RuntimeError: Error(s) in loading state_dict for UNet:
    size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 21, 3, 3, 3]).
    size mismatch for model.2.0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.weight: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_mean: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_var: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.1.conv.unit0.conv.weight: copying a param with shape torch.Size([14, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([21, 21, 3, 3, 3]).
    size mismatch for model.2.1.conv.unit0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
Loading dataset: 100%|██████████| 2/2 [00:36<00:00, 17.95s/it]
Loading dataset: 100%|██████████| 2/2 [00:36<00:00, 18.22s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
2023-01-23 11:34:01,725 - 1 - Records for Training: 2
2023-01-23 11:34:01,735 - Engine run resuming from iteration 0, epoch 0 until 50 epochs
Loading dataset: 100%|██████████| 2/2 [00:36<00:00, 17.79s/it]
Loading dataset: 100%|██████████| 2/2 [00:36<00:00, 18.26s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
2023-01-23 11:34:01,753 - 4 - Records for Training: 2
2023-01-23 11:34:01,762 - Engine run resuming from iteration 0, epoch 0 until 50 epochs
2023-01-23 11:34:01,800 - Engine run is terminating due to exception: Error(s) in loading state_dict for UNet:
    size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 21, 3, 3, 3]).
    size mismatch for model.2.0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.weight: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_mean: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_var: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.1.conv.unit0.conv.weight: copying a param with shape torch.Size([14, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([21, 21, 3, 3, 3]).
    size mismatch for model.2.1.conv.unit0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
2023-01-23 11:34:01,818 - Engine run is terminating due to exception: Error(s) in loading state_dict for UNet:
    size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 21, 3, 3, 3]).
    size mismatch for model.2.0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.weight: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_mean: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_var: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.1.conv.unit0.conv.weight: copying a param with shape torch.Size([14, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([21, 21, 3, 3, 3]).
    size mismatch for model.2.1.conv.unit0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
Loading dataset: 100%|██████████| 2/2 [00:37<00:00, 18.18s/it]
Loading dataset: 100%|██████████| 2/2 [00:37<00:00, 18.67s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
2023-01-23 11:34:02,583 - 5 - Records for Training: 2
2023-01-23 11:34:02,593 - Engine run resuming from iteration 0, epoch 0 until 50 epochs
2023-01-23 11:34:02,670 - Engine run is terminating due to exception: Error(s) in loading state_dict for UNet:
    size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 21, 3, 3, 3]).
    size mismatch for model.2.0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.weight: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_mean: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_var: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.1.conv.unit0.conv.weight: copying a param with shape torch.Size([14, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([21, 21, 3, 3, 3]).
    size mismatch for model.2.1.conv.unit0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
Loading dataset: 100%|██████████| 2/2 [00:37<00:00, 18.26s/it]
Loading dataset: 100%|██████████| 2/2 [00:37<00:00, 18.74s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
2023-01-23 11:34:02,767 - 3 - Records for Training: 2
2023-01-23 11:34:02,776 - Engine run resuming from iteration 0, epoch 0 until 50 epochs
2023-01-23 11:34:02,906 - Engine run is terminating due to exception: Error(s) in loading state_dict for UNet:
    size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 21, 3, 3, 3]).
    size mismatch for model.2.0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.weight: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_mean: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_var: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.1.conv.unit0.conv.weight: copying a param with shape torch.Size([14, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([21, 21, 3, 3, 3]).
    size mismatch for model.2.1.conv.unit0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
Loading dataset: 100%|██████████| 2/2 [00:38<00:00, 18.74s/it]
Loading dataset: 100%|██████████| 2/2 [00:38<00:00, 19.19s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
2023-01-23 11:34:03,583 - 6 - Records for Training: 2
2023-01-23 11:34:03,595 - Engine run resuming from iteration 0, epoch 0 until 50 epochs
2023-01-23 11:34:03,952 - Engine run is terminating due to exception: Error(s) in loading state_dict for UNet:
    size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 21, 3, 3, 3]).
    size mismatch for model.2.0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.weight: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_mean: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_var: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.1.conv.unit0.conv.weight: copying a param with shape torch.Size([14, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([21, 21, 3, 3, 3]).
    size mismatch for model.2.1.conv.unit0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
Traceback (most recent call last):
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 132, in <module>
    run_main()
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 117, in run_main
    result = a.train(request)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 421, in train
    result = task(request, self.datastore())
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/tasks/train/basic_train.py", line 392, in __call__
    torch.multiprocessing.spawn(main_worker, nprocs=world_size, args=(world_size, req, datalist, self))
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 7 terminated with the following error:
Traceback (most recent call last):
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/tasks/train/basic_train.py", line 625, in main_worker
    task.train(rank, world_size, request, datalist)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/tasks/train/basic_train.py", line 458, in train
    context.trainer.run()
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/engines/trainer.py", line 53, in run
    super().run()
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/engines/workflow.py", line 278, in run
    super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/engine/engine.py", line 892, in run
    return self._internal_run()
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/engine/engine.py", line 935, in _internal_run
    return next(self._internal_run_generator)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/engine/engine.py", line 993, in _internal_run_as_gen
    self._handle_exception(e)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/engine/engine.py", line 638, in _handle_exception
    raise e
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/engine/engine.py", line 946, in _internal_run_as_gen
    self._fire_event(Events.STARTED)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/engine/engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/handlers/checkpoint_loader.py", line 128, in __call__
    Checkpoint.load_objects(to_load=self.load_dict, checkpoint=checkpoint, strict=self.strict)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/handlers/checkpoint.py", line 635, in load_objects
    _load_object(obj, checkpoint_obj[k])
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/ignite/handlers/checkpoint.py", line 620, in _load_object
    obj.load_state_dict(chkpt_obj, strict=is_state_dict_strict)
  File "/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1667, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNet:
    size mismatch for model.2.0.conv.weight: copying a param with shape torch.Size([32, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 21, 3, 3, 3]).
    size mismatch for model.2.0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.weight: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_mean: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.0.adn.N.running_var: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
    size mismatch for model.2.1.conv.unit0.conv.weight: copying a param with shape torch.Size([14, 14, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([21, 21, 3, 3, 3]).
    size mismatch for model.2.1.conv.unit0.conv.bias: copying a param with shape torch.Size([14]) from checkpoint, the shape in current model is torch.Size([21]).
/home/vincent.ochs/miniconda3/envs/monailabel-env/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
[2023-01-23 11:34:05,240] [1108528] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:77) - Return code: 1

I addition to that I am wondering where my model gets stored after it got trained and also the masks
Are the masks stored as nifti files anywhere?

Happy to receive any feedback.
Also if you need additional information please let me know!

Thank you!

[diazandr3s]

Hi @vincentochs,

It seems the app is trying to use a pretrained model and it fails because it doesn't match the number of labels.
If you'd like to train a model from scratch, remove the pt files and folders inside the radiology/model folder within the radiology app, and set to false the use of the pretrained model in the configs file (i.e. if using the segmentation model: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/configs/segmentation.py#L59)

Regarding this:

I addition to that I am wondering where my model gets stored after it got trained and also the masks Are the masks stored as nifti files anywhere?

The model is stored inside the radiology/model folder

Masks should be located within YOUR_DATASET/labels/final/

Hope this helps,

[vincentochs]

Hello @diazandr3s . thank you very much! That helped a lot and it is running now!!!

One follow up question if I am allowed to:
-Is it using a UNet in the background?
-Is it possible to use this UNet later then for segmentation, or should I train another model after labeling is done ?

-Would you recommend to increase Epoch Size in order to manually label as little as possible? Or which parameter should I change in order to increase computational burden but therefore decrease manual labeling work?

Thank you once again!

[diazandr3s]

Hi @vincentochs,

-Is it using a UNet in the background?

Yes, you can see the architecture here: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/configs/segmentation.py#L69

-Is it possible to use this UNet later then for segmentation, or should I train another model after labeling is done ?

Yes, you should have the weights in a .pt file in the radiology/model folder once it is trained

-Would you recommend to increase Epoch Size in order to manually label as little as possible? Or which parameter should I change in order to increase computational burden but therefore decrease manual labeling work?

Yes, more training can help as well as more training samples.

[vincentochs]

Dear @diazandr3s thank you very much!

Is it possible to stop epochs after it does not get any better? (like setting "early_stop_patience" to 10 ?)

Is it possible to use the saved .pt file to visualize results already after it got trained?

What if I change the epoch size from one training to another? Does the overall model starts from the beginning? Or does it create a 2nd .pt file with the results? So in other words -> Is this dynamic?

Do I need to take care of data augmentation? Since it is an UNet I am wondering if this needs to be calibrated by me?

Is a validation set is getting trained somewhere so that I can check when to stop with epoch size?