Bundle Adjusted Gaussian Avatars Deblurring

Muyao Niu ^1,2   Yifan Zhan^1,2   Qingtian Zhu²   Zhuoxiao Li²   Wei Wang¹  
Zhihang Zhong^1,†   Xiao Sun^1,†   Yinqiang Zheng²  

¹Shanghai Artificial Intelligence Laboratory   ²The University of Tokyo

^†Corresponding Authors  

---

Stay tuned. Feel free to contact me for bugs or missing files.

Setup Procedures

Python Environment

conda create -n baga python==3.8 -y
conda activate baga
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install submodules/diff-gaussian-rasterization/
pip install submodules/simple-knn/
pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
pip install -r requirement.txt

SMPL files

Register and download SMPL models here and put the downloaded models into the folder assets. Only the neutral one is needed. The folder structure should look like

./
└── assets/
    ├── SMPL_NEUTRAL.pkl

Dataset

We contribute synthetic and real datasets for evaluating blur-aware 3DGS human avatar synthesis techniques.

Synthetic Dataset

For the synthetic dataset, due to the aggreement of ZJU-MoCap, we cannot re-distribute the sharp data of ZJU-MoCap. So you have to download the original dataset, and follow the following steps to construct the final synthetic dataset using our scripts:

1. Download the blurry frames and the calibrations from here and unzip it to ./data/BlurZJU.
2. Follow the procedure here to download ZJU-MoCap (refined version). Unzip and put the six scenes (my_377, my_386, my_387, my_392, my_393, my_394) to ./data/ZJU-MoCap-Refine (If you get scenes starting with CoreView instead of my, then you have downloaded the original ZJU-MoCap, not the Refined version).
3. Run python rearrange_zju.py to re-arrange the dataset.

Real Dataset (BS-Human)

Download the real dataset from this link and unzip them to the ./data directory.

Training

Synthetic dataset

chmod 777 train_BlurZJU.sh
bash train_BlurZJU.sh

Real dataset

chmod 777 train_BSHuman.sh
bash train_BSHuman.sh

Acknowledgments

We appreciate gaussian-splatting, GauHuman, and GSM for their wonderful work and code implementation. We would also like to deeply express our gratitude to the release of NeuralBody (as well as the ZJU-MoCap dataset) and EasyMocap which we use to calibrate our dataset.