Skip to content

Latest commit

 

History

History
174 lines (135 loc) · 7.5 KB

README.md

File metadata and controls

174 lines (135 loc) · 7.5 KB

[CVPR2025] Omni-Scene⚡️: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction

Dongxu Wei  ·  Zhiqi Li  ·  Peidong Liu*

Paper | Project Page

This as an official implementation of our CVPR 2025 paper Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction, based on the mmdet3d framework and accelerate library. In this repository, we provide our code, along with our pre-processed nuScenes dataset, which is specially reformulated for ego-centric scene reconstruction.

News

  • 2025/2/27: Our paper is accepted by CVPR 2025!
  • 2025/3/3: Code and data are now available. If you find this repository useful, please give us a star🌟!

TODO

  • Upload code and data
  • Add tutorial for 3D generation
  • Add code and documents for Waymo dataset

Demo (3D Reconstruction)

Reconstruct scenes with 3D Gaussians given 6-view images from nuScenes in a feed-forward manner:

recon_examples_normal.mp4

Demo (3D Generation)

Generate scenes with 3D Gaussians by combining our method with 2D multi-view diffusion model MagicDrive:

synth_examples.mp4

Get started

1. Installation

# (Optional) create a fresh conda env
conda create --name omniscene -y "python==3.10"
conda activate omniscene

# install dependencies
pip install --upgrade pip setuptools

## install pytorch (CUDA 11.8)
pip install "torch==2.1.0+cu118" "torchvision==0.16.0+cu118" torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118
## install pytorch (CUDA 12.1)
pip install "torch==2.1.0+cu121" "torchvision==0.16.0+cu121" torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121

## install 3DGS rasterizer (w/ depth)
git clone --recursive https://github.com/ashawkey/diff-gaussian-rasterization
pip install diff-gaussian-rasterization

## common dependencies
pip install -r requirements.txt

## compile mmcv-2.1.0 and install mmdet3d-1.4.0
pip install -U openmim
pip install mmengine
pip install ninja psutil
git clone https://github.com/open-mmlab/mmcv.git
git checkout v2.1.0
MAX_JOBS=16 MMCV_WITH_OPS=1 FORCE_CUDA=1 TORCH_CUDA_ARCH_LIST="8.9+PTX" pip install -e . -v
mim install 'mmdet>=3.0.0'
mim install 'mmdet3d==1.4.0'

You can refer to MMLab documents for details about mmcv and mmdet3d installation.

2. Prepare the dataset

We have provided a pre-processed dataset on OneDrive, and BaiduYun (extract code: qbck). You can download and extract the compressed data files as follows.

cat dataset_omniscene_part* > dataset_omniscene.tar
tar -xvf dataset_omniscene.tar
rm -rf *.tar
rm -rf *part*
mv dataset_omniscene {ROOT}/data/nuScenes

Put the extracted files under {ROOT}/data, and the data should be structured like this:

{ROOT}/data/nuScenes
├── samples_small
├── samples_dpt_small
├── samples_dptm_small
├── samples_param_small
├── sweeps_small
├── sweeps_dpt_small
├── sweeps_dptm_small
├── sweeps_param_small
├── interp_12Hz_trainval

3. Training

The training script is as follows. We have released our pre-trained weights here.

accelerate launch --config-file accelerate_config.yaml train.py \
    --py-config configs/OmniScene/omni_gs_nusc_novelview_r50_224x400.py \
    --work-dir workdirs/omni_gs_nusc_novelview_r50_224x400 \
    --resume-from path/to/checkpoints

where

  • --config-file accelerate_config.yaml is the relative path of accelrate configuration file;
  • --py-config configs/OmniScene/omni_gs_nusc_novelview_r50_224x400.py" is the relative path of Omni-Scene's configuration file;
  • --work-dir is the relative path of experiment work directory. We save logs, checkpoints, visualizations, plys here;
  • (optional) --resume-from is the relative path of checkpoints that you want to resume from. You should delete this argument for training from scratch.

Note: >=2 A100 GPUs are required to run the training of our full method.

4. Evaluation

The evaluation script is as follows.

accelerate launch --config-file accelerate_config.yaml evaluate.py \
    --py-config configs/OmniScene/omni_gs_nusc_novelview_r50_224x400.py \
    --output-dir outputs/omni_gs_nusc_novelview_r50_224x400 \
    --load-from checkpoints/checkpoint-100000

where

  • --config-file accelerate_config.yaml is the relative path of accelrate configuration file;
  • --py-config configs/OmniScene/omni_gs_nusc_novelview_r50_224x400.py" is the relative path of Omni-Scene's configuration file;
  • --output-dir is the relative path of output directory. We save the results and visualizations here.
  • --load-from is the relative path of model weights that you want to evaluate.

Note: >=1 A100 GPUs are required to run the evaluation of our full method.

5. Running demo

This command will generate and save 360 degree exploring videos for the reconstructed 3D scenes:

accelerate launch --config-file accelerate_config.yaml demo.py \
    --py-config configs/OmniScene/omni_gs_nusc_novelview_r50_224x400.py \
    --output-dir outputs/omni_gs_nusc_novelview_r50_224x400_vis \
    --load-from checkpoints/checkpoint-100000

where

  • --config-file accelerate_config.yaml is the relative path of accelrate configuration file;
  • --py-config configs/OmniScene/omni_gs_nusc_novelview_r50_224x400.py" is the relative path of Omni-Scene's configuration file;
  • --output-dir is the relative path of output directory. We save the rendered videos here.
  • --load-from is the relative path of model weights that you want to use.

Note: >=1 A100 GPUs are required to run the demo of our full method.

Citation

If you find this useful, please consider citing:

@inproceedings{wei2024omniscene,
    author = {Wei, Dongxu and Li, Zhiqi and Liu, Peidong},
    title = {Omni-scene: omni-gaussian representation for ego-centric sparse-view scene reconstruction},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year = {2025}
}

Acknowledgments

Omni-Scene is based on mmdetection3d. It is also greatly inspired by the following outstanding contributions to the open-source community: TPVFormer, BEVFormer, MVSplat, pixelSplat, Metric3D-v2 and MagicDrive.