LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

Project page | Paper | Video | Huggingface

Shuai Yang*, Jing Tan*, Mengchen Zhang, Tong Wu✉️, Yixuan Li, Gordon Wetzstein, Ziwei Liu, Dahua Lin✉️

*Equal Contribution, ✉️Corresponding author

🔧 Installation:

• Prepare Environment

conda create -n layerpano3d python==3.9
conda activate layerpano3d
pip install -r requirements.txt
pip install timm==0.4.12 --no-deps
pip install xformers==0.0.27.post2 --index-url https://download.pytorch.org/whl/cu118
pip install -e submodules/diff-gaussian-rasterization
pip install -e submodules/simple-knn

• Install PyTorch. We have tested on torch2.4.0+cu118, but other versions should also work fine.

pip install torch==2.4.0 torchvision==0.19.0  --index-url https://download.pytorch.org/whl/cu118

• Install 360monodepth. This might be tricky. Do it step by step. Ask in Issue if you have any problems.

#Important step here
Modify `submodules/360monodepth/code/cpp/CMakeConfig.txt` Line 65, Change the path of numpy to your own.

sudo apt-get install libgtest-dev libeigen3-dev libboost-all-dev libopencv-dev libatlas-base-dev
sudo apt-get install liblapack-dev libsuitesparse-dev libcxsparse3 libgflags-dev libgoogle-glog-dev libgtest-dev
conda install -c conda-forge libstdcxx-ng=12  

# pybind11
cd submodules/360monodepth/code/cpp/3rd_party
git clone https://github.com/pybind/pybind11.git 
cd pybind11 && mkdir build && cd build
cmake .. && make -j8 
sudo make install
cd ../../ 

# ceres-solver
git clone -b 1.14.0 https://github.com/ceres-solver/ceres-solver
cd ceres-solver && mkdir build && cd build
cmake .. && make -j8 
sudo make install 
cd ../../../  

# instaOmniDepth
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release  ..
make -j8 
cd ../python
python setup.py build
python setup.py bdist_wheel 
pip install dist/instaOmniDepth-0.1.0-cp39-cp39-linux_x86_64.whl # if failed, please check your file version in dist/

• Checkpoints Download:
  • Download the Panorama LoRA checkpoint and put it in the checkpoints folder. We will release more powerful and better panorama_lora_v2.safetensors() in the future, now version-1 is avaliable in huggingface.
  • Download the Lama checkpoint and put it in the checkpoints folder.
  • Download the SAM model ViT-H checkpoints and put it in the checkpoints folder.
  • Download the Depth-Anything-V2-Large checkpoints and put it in the checkpoints folder.
  • Download the Infusion checkpoint and put it in the checkpoints/Infusion folder.
  • [Optional] Download the PASD related checkpoints. If you do not need SR panorama for better visual quality, this step can be skip.
    • Download SD1.5 models from huggingface and put them into checkpoints/pasd/stable-diffusion-v1-5.
    • Download PASD pre-trained models pasd and put them into checkpoints/pasd/checkpoint-100000.

🌌 Inference:

Quick start: one-step running bash. From a single text prompt to generates full-view, explorable panoramic 3D scene.

huggingface-cli login
bash run.sh

If you want to try or debug each stage in our pipeline, we also provide a complete running results (including each step).

🌆 Rendering:

In the generated 3DGS panoramic scene, the resulting folder contains files formatted as gsplat_layer{id}.ply, where id represents the index of the layer. The number of layers varies across different cases. Notably, the 3dgs file corresponding to the maximum id value represents the final 3D scene result.

bash render.sh

🌉 Visualize .ply files

We recommend use @playcanvas's Super-Splat project (Live demo).

📧 Contact Us

Shuai Yang: [email protected]
Jing Tan: [email protected]
Tong Wu: [email protected]

✒️ Citation

If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝

@article{yang2024layerpano3d,
  title={LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation},
  author={Yang, Shuai and Tan, Jing and Zhang, Mengchen and Wu, Tong and Li, Yixuan and Wetzstein, Gordon and Liu, Ziwei and Lin, Dahua},
  journal={arXiv preprint arXiv:2408.13252},
  year={2024}
}