After modifying the transforms in trainers/deepedit, the model cannot learn. #907

Minxiangliu · 2022-08-04T02:14:12Z

Minxiangliu
Aug 4, 2022

My version: from docker projectmonai/monailabel:0.4.2

3D Slicer version: 5.0.3

Server Executio: monailabel start_server --app apps/radiology --studies HNC-datasets --conf models deepedit

After I executed monailabel apps --download --name radiology --output apps, I modified part of the following two files:

path: apps/radiology/lib/configs/deepedit.py

# Multilabel
self.labels = {
      "cancer": 1,
      "transfer": 2,
      "background": 0
 }

self.conf.get("use_pretrained_model", "false")

path: apps/radiology/lib/trainers/deepedit.py

def train_pre_transforms(self, context: Context):
        return [
            LoadImaged(keys=("image", "label"), reader="ITKReader"),
            NormalizeLabelsInDatasetd(keys="label", label_names=self._labels),
            #========================================================================
            EnsureChannelFirstd(keys=("image", "label")),
            Orientationd(keys=["image", "label"], axcodes="RAS"),
            Spacingd(keys=['image','label'], pixdim=(4.,4.,2.57), mode=('bilinear', 'nearest')),
            Rotate90d(keys=['image','label'], spatial_axes=(0,1)),
            Flipd(keys=['image','label'], spatial_axis=1),
            ScaleIntensityRanged(keys="image", a_min=0, a_max=15000, b_min=0.0, b_max=1.0, clip=True),
            # Resized(keys=("image", "label"), spatial_size=self.spatial_size, mode=("area", "nearest")),
            CropForegroundd(keys=['image','label'],
                    source_key='image', 
                    select_fn=lambda x:x>0.05, 
                    margin=0, 
                    return_coords=False),
            SpatialPadd(keys=['image','label'], spatial_size=self.spatial_size),
            #========================================================================
            RandScaleIntensityd(keys=['image'], factors=0.2, prob=1.0),
            RandShiftIntensityd(keys=['image'], offsets=(-0.8, 0.8), prob=1.0),
            RandSpatialCropd(keys=['image', 'label'], roi_size=[64,64,32], random_size=False),
            RandFlipd(keys=("image", "label"), spatial_axis=[0], prob=0.10),
            RandFlipd(keys=("image", "label"), spatial_axis=[1], prob=0.10),
            RandFlipd(keys=("image", "label"), spatial_axis=[2], prob=0.10),
            RandRotate90d(keys=("image", "label"), prob=0.10, max_k=3),
            #========================================================================
            # Transforms for click simulation
            FindAllValidSlicesMissingLabelsd(keys="label", sids="sids"),
            AddInitialSeedPointMissingLabelsd(keys="label", guidance="guidance", sids="sids"),
            AddGuidanceSignalDeepEditd(keys="image", guidance="guidance", number_intensity_ch=self.number_intensity_ch),
            #========================================================================
            ToTensord(keys=("image", "label")),
            SelectItemsd(keys=("image", "label", "guidance", "label_names")),
        ]

def val_pre_transforms(self, context: Context):
        return [
            LoadImaged(keys=("image", "label"), reader="ITKReader"),
            NormalizeLabelsInDatasetd(keys="label", label_names=self._labels),
            #========================================================================
            EnsureChannelFirstd(keys=("image", "label")),
            Orientationd(keys=["image", "label"], axcodes="RAS"),
            Spacingd(keys=['image','label'], pixdim=(4.,4.,2.57), mode=('bilinear', 'nearest')),
            Rotate90d(keys=['image','label'], spatial_axes=(0,1)),
            Flipd(keys=['image','label'], spatial_axis=1),
            ScaleIntensityRanged(keys="image", a_min=0, a_max=15000, b_min=0.0, b_max=1.0, clip=True),
            # Resized(keys=("image", "label"), spatial_size=self.spatial_size, mode=("area", "nearest")),
            CropForegroundd(keys=['image','label'],
                    source_key='image', 
                    select_fn=lambda x:x>0.05, 
                    margin=0, 
                    return_coords=False),
            SpatialPadd(keys=['image','label'], spatial_size=self.spatial_size),
            #========================================================================
            FindAllValidSlicesMissingLabelsd(keys="label", sids="sids"),
            AddInitialSeedPointMissingLabelsd(keys="label", guidance="guidance", sids="sids"),
            AddGuidanceSignalDeepEditd(keys="image", guidance="guidance", number_intensity_ch=self.number_intensity_ch),
            #========================================================================
            ToTensord(keys=("image", "label")),
            SelectItemsd(keys=("image", "label", "guidance", "label_names")),
        ]

Problem Statement:
I have prepared 4 sets of PET CT images and labels, and tried the current model learning effect, but found that if I use Resized at the end of transforms, the model can be trained normally, but after changing to CropForegroundd and SpatialPadd, I cannot learn anything (dice score is 0 ).

Result:
If use:

self.spatial_size = (128,128,64)
EnsureChannelFirstd(keys=("image", "label")),
Orientationd(keys=["image", "label"], axcodes="RAS"),
Spacingd(keys=['image','label'], pixdim=(4.,4.,2.57), mode=('bilinear', 'nearest')),
Rotate90d(keys=['image','label'], spatial_axes=(0,1)),
Flipd(keys=['image','label'], spatial_axis=1),
ScaleIntensityRanged(keys="image", a_min=0, a_max=15000, b_min=0.0, b_max=1.0, clip=True),
CropForegroundd(keys=['image','label'],
        source_key='image', 
        select_fn=lambda x:x>0.05, 
        margin=0, 
        return_coords=False),
SpatialPadd(keys=['image','label'], spatial_size=self.spatial_size),

If use:

self.spatial_size = (128,128,64)
EnsureChannelFirstd(keys=("image", "label")),
Orientationd(keys=["image", "label"], axcodes="RAS"),
Spacingd(keys=['image','label'], pixdim=(4.,4.,2.57), mode=('bilinear', 'nearest')),
Rotate90d(keys=['image','label'], spatial_axes=(0,1)),
Flipd(keys=['image','label'], spatial_axis=1),
ScaleIntensityRanged(keys="image", a_min=0, a_max=15000, b_min=0.0, b_max=1.0, clip=True),
Resized(keys=("image", "label"), spatial_size=self.spatial_size, mode=("area", "nearest")),

Verify Transforms:
In order to verify whether my transforms is normal, I implemented it directly in monai:

_ImageNii = [r"C:\Users\hp\Desktop\HNC\HNC-datasets\NPC_AI_Detectionl_01.nii.gz"]
_MasNiik = [r"C:\Users\hp\Desktop\HNC\HNC-datasets\labels\final\NPC_AI_Detectionl_01.nii.gz"]

files = [{'image':x, 'label':y} for x,y in zip(_ImageNii, _MasNiik)]
tran = Compose([
    LoadImaged(keys=['image','label']),
    EnsureChannelFirstd(keys=['image','label']),
    Orientationd(keys=['image','label'], axcodes='RAS'),
    Spacingd(keys=['image','label'], pixdim=(4.,4.,2.57), mode=('bilinear','nearest')),
    Rotate90d(keys=['image','label'], spatial_axes=(0,1)),
    Flipd(keys=['image','label'], spatial_axis=1),
    ScaleIntensityRanged(
            keys=['image'], a_min=0, a_max=15000,
            b_min=0.0, b_max=1.0, clip=True,
    ),
    CropForegroundd(keys=['image','label'],
                    source_key='image', 
                    select_fn=lambda x:x>0.05, 
                    margin=0, 
                    return_coords=False
                    ),
    SpatialPadd(keys=['image','label'], spatial_size=(128,128,64)),
    ])

Can someone help me with the problem? Thanks in advance.
Here are the apps and datasets used.

Answered by tangy5

Aug 6, 2022

Hi @Minxiangliu , thanks for reporting this issue. After reproducing the app and your transformations, we found the NormalizeLabelsInDatasetd will not modify the data meta information correctly, but the following transformation after EnsureChannelFirstd will be handled as MetaTensor, where the image meta information are stored. If the NormalizeLabelsInDatasetd is applied in the beginning, the meta information will be lost and resulting incorrect transformations.

For now, the solution is to move the NormalizeLabelsInDatasetd after all MONAI image transformations are done (As bellow).
I've tested it using your app and data, it gives me the results:

Result: {"total_epochs": 50, "total_iterat…

View full answer

tangy5 · 2022-08-05T06:16:15Z

tangy5
Aug 5, 2022
Maintainer

Hi @Minxiangliu Let me try to reproduce the results. Is the above image the demo using CropForeground and SpatialPadd? The image looks normal. I will try figure this out and get reply later.

0 replies

tangy5 · 2022-08-06T05:42:27Z

tangy5
Aug 6, 2022
Maintainer

Hi @Minxiangliu , thanks for reporting this issue. After reproducing the app and your transformations, we found the NormalizeLabelsInDatasetd will not modify the data meta information correctly, but the following transformation after EnsureChannelFirstd will be handled as MetaTensor, where the image meta information are stored. If the NormalizeLabelsInDatasetd is applied in the beginning, the meta information will be lost and resulting incorrect transformations.

For now, the solution is to move the NormalizeLabelsInDatasetd after all MONAI image transformations are done (As bellow).
I've tested it using your app and data, it gives me the results:

Result: {"total_epochs": 50, "total_iterations": 1, "epoch": 50, "start_ts": 1659763878, "total_time": "0:00:15", "best_metric": 0.4035555422306061, "train": {"metrics": {"train_dice": 0.0, "cancer_dice": 0.0, "transfer_dice": 0.0}, "key_metric_name": "train_dice", "best_metric": 0.4035555422306061, "best_metric_epoch": 44}}


            LoadImaged(keys=("image", "label"), reader="ITKReader"),
            EnsureChannelFirstd(keys=("image", "label")),
            #========================================================================
            Orientationd(keys=("image", "label"), axcodes="RAS"),
            Spacingd(keys=("image","label"), pixdim=(4.,4.,2.57), mode=('bilinear', 'nearest')),
            Rotate90d(keys=("image","label"), spatial_axes=(0,1)),
            Flipd(keys=("image","label"), spatial_axis=1),
            ScaleIntensityRanged(keys=("image"), a_min=0, a_max=15000, b_min=0.0, b_max=1.0, clip=True),
            # Resized(keys=("image", "label"), spatial_size=self.spatial_size, mode=("area", "nearest")),
            CropForegroundd(keys=("image","label"),
                    source_key="image", 
                    select_fn=lambda x:x>0.01, 
                    margin=0, 
                    return_coords=False),
            SpatialPadd(keys=("image","label"), spatial_size=self.spatial_size),
            #========================================================================
            RandScaleIntensityd(keys=['image'], factors=0.2, prob=1.0),
            RandShiftIntensityd(keys=['image'], offsets=(-0.8, 0.8), prob=1.0),
            RandSpatialCropd(keys=['image', 'label'], roi_size=[64,64,32], random_size=False),
            RandFlipd(keys=("image", "label"), spatial_axis=[0], prob=0.10),
            RandFlipd(keys=("image", "label"), spatial_axis=[1], prob=0.10),
            RandFlipd(keys=("image", "label"), spatial_axis=[2], prob=0.10),
            RandRotate90d(keys=("image", "label"), prob=0.10, max_k=3),
            NormalizeLabelsInDatasetd(keys=("label"), label_names=self._labels),
            #========================================================================
            # Transforms for click simulation
            FindAllValidSlicesMissingLabelsd(keys="label", sids="sids"),
            AddInitialSeedPointMissingLabelsd(keys="label", guidance="guidance", sids="sids"),
            AddGuidanceSignalDeepEditd(keys="image", guidance="guidance", number_intensity_ch=self.number_intensity_ch),
            #========================================================================
            ToTensord(keys=("image", "label")),
            SelectItemsd(keys=("image", "label", "guidance", "label_names")),

We will continue look into this issue and see whether the NormalizeLabelsInDatasetd need further modification to be compatible with MetaTensor.

Thanks

1 reply

Minxiangliu Aug 7, 2022
Author

Thanks for your help, looking forward to this great tool being even more powerful!

diazandr3s · 2022-08-07T22:29:39Z

diazandr3s
Aug 7, 2022
Maintainer

Thanks for reporting this issue, @Minxiangliu.

Changing the pre-transforms for the DeepEdit model is tricky as it involves click simulation (https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/deepedit.py#L114-L117). Cropping the image and label by the foreground deletes most of the background and it could affect performance. It is not always the case but that should be considered.

The transform NormalizeLabelsInDatasetd ensures that the label numbering is consistent among all the training samples. Especially for the training samples that don't have all the labels declared in the configs file. Actually, this transform shouldn't affect performance if you make sure all the images have the labels you declared and the same label numbering (i.e. cancer is always 1 and transfer is always 2).

If you'd like to have more flexibility in the pre-transforms, I'd recommend you the segmentation model. Here is an example: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/localization_spine.py#L72-L84

Hope that helps,

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After modifying the transforms in trainers/deepedit, the model cannot learn. #907

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 1 reply

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

After modifying the transforms in trainers/deepedit, the model cannot learn. #907

Minxiangliu Aug 4, 2022

Replies: 3 comments · 1 reply

tangy5 Aug 5, 2022 Maintainer

tangy5 Aug 6, 2022 Maintainer

Minxiangliu Aug 7, 2022 Author

diazandr3s Aug 7, 2022 Maintainer

Minxiangliu
Aug 4, 2022

Replies: 3 comments 1 reply

tangy5
Aug 5, 2022
Maintainer

tangy5
Aug 6, 2022
Maintainer

Minxiangliu Aug 7, 2022
Author

diazandr3s
Aug 7, 2022
Maintainer