Application of 3D Gaussian Splatting for Cinematic Anatomy on Consumer Class Devices

Simon Niedermayr¹, Christoph Neuhauser¹, Kaloian Petkov², Klaus Engel², Rüdiger Westermann^1
1Technical University of Munich, ²Siemens Healthineers

Webpage | arXiv

Abstract

Interactive photorealistic visualization of 3D anatomy (i.e., Cinematic Anatomy) is used in medical education to explain the structure of the human body. It is currently restricted to frontal teaching scenarios, where the demonstrator needs a powerful GPU and high-speed access to a large storage device where the dataset is hosted. We demonstrate the use of novel view synthesis via compressed 3D Gaussian splatting to overcome this restriction and to enable students to perform cinematic anatomy on lightweight mobile devices and in virtual reality environments. We present an automatic approach for finding a set of images that captures all potentially seen structures in the data. By mixing closeup views with images from a distance, the splat representation can recover structures up to the voxel resolution. The use of Mip-Splatting enables smooth transitions when the focal length is increased. Even for GB datasets, the final renderable representation can usually be compressed to less than 70 MB, enabling interactive rendering on low-end devices using rasterization.

Citation

If you find our work useful, please cite:

@misc{niedermayr2024novel,
    title={Application of 3D Gaussian Splatting for Cinematic Anatomy on Consumer Class Devices},
    author={Simon Niedermayr and Christoph Neuhauser and Kaloian Petkov and Klaus Engel and Rüdiger Westermann},
    year={2024},
    eprint={2404.11285},
    archivePrefix={arXiv},
    primaryClass={cs.GR}
}

Installation

Best to be installed with anaconda or miniconda:

1. Clone the repo:

git clone https://https://github.com/KeKsBoTer/cinematic-gaussians.git --recursive
cd cinematic-gaussians

2. Create environment and install dependencies

conda create -n cin3dgs python=3.12 -y
conda activate cin3dgs

Training

python train.py
    -s <scene folder> \\
    -m <model output folder> \\
    --eval \\
    --test_iterations 7000 15000 30000 \\
    --densify_grad_threshold 0.00005 \\
    --save_iterations 30000 \\

If you run into VRAM issues try increasing the densify_grad_threshold parameter to reduce the number of Gaussians.

Compression

To compress the model with our method run the compression script on the reconstructed scenes:

python compress.py \\
    -m <training_output_folder> \\
    --eval \\
    --output_vq <compression_output_folder> \\
    --load_iteration 30000

This script will also evaluate the scene on the training and test images and report PSNR and SSIM.

View Selection

The code for the view selection can be found here