Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

---

Zhitong Gao · Bingnan Li · Mathieu Salzmann · Xuming He

[Arxiv] [Poster] [Slides] [Video]

NeurIPS 2024 Proceedings

Figure 1: The overview of the proposed method.

---

Abstract

In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor OOD detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization.

---

Environment Setup

conda env create -f environment.yml 

conda activate MultiShiftSeg
git clone https://github.com/facebookresearch/detectron2.git
pip install -e detectron2
pip install git+https://github.com/cocodataset/panopticapi.git
cd lib/network/mask2former/modeling/pixel_decoder/ops
sh make.sh

Data Preparation

datasets
├── cityscapes                      
├── road_anomaly
│   ├── original
│   └── labels
├── dataset_AnomalyTrack            #RA21
│   ├── images
│   └── labels_masks
├── dataset_ObstacleTrack           #RO21
│   ├── images
│   ├── image-sources.txt
│   └── labels_masks
├── MUAD_challenge                  
│   └── test_sets
│       └── test_OOD
│           ├── leftImg8bit
│           └── leftLabel
├── acdc_ood                        #ACDC-POC
│   ├── gt_trainval
│   └── rgb_anon_trainvaltest
├── fs_LostAndFound                 
│   ├── original
│   └── labels 
└── fs_static
    ├── original
    ├── labels 
    └── match.npy

Generated data can be downloaded from Google Drive or Hugging Face. To generate the data, please refer to the Generation Instruction.

For more detailed instructions, please refer to the Dataset Instruction.

Checkpoint

	RoadAnomaly	RoadAnomaly	RoadAnomaly	SMIYC-RA21	SMIYC-RA21	SMIYC-RO21	SMIYC-RO21	Weights
Method	AUC	AP	FPR	AP	FPR	AP	FPR
DeepLab v3+	96.40	74.60	16.08	88.06	8.21	90.71	0.26	Google Drive or Hugging Face
Mask2Former	97.94	90.17	7.54	91.92	7.94	95.29	0.07	Google Drive or Hugging Face

Training

Coming Soon.

Evaluation

Coming Soon.

BibTeX

@inproceedings{
gao2024generalize,
title={Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts},
author={Zhitong Gao and Bingnan Li and Mathieu Salzmann and Xuming He},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=h0rbjHyWoa}
}