Skip to content

Latest commit

 

History

History
172 lines (137 loc) · 5.85 KB

README.md

File metadata and controls

172 lines (137 loc) · 5.85 KB

RNb-NeuS2

Robin Bruneau★ · Baptiste Brument★ · Yvain Quéau
Jean Mélou · François Lauze · Jean-Denis Durou · Lilian Calvet

★ corresponding author

Paper PDF Project Page

Welcome to the official CUDA implementation of RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction.

Table of Contents

Installation

Follow the Instant-NGP instructions for requirements and compilation. NeuS2 installation steps are similar.

Clone the repository and its submodules:

git clone https://github.com/RobinBruneau/RNb-NeuS2/
cd RNb-NeuS2

Build the project using CMake:

cmake . -B build
cmake --build build --config RelWithDebInfo -j

Ensure you have Python and the following libraries installed:

  • Numpy
  • Scipy
  • Argparse
  • Json
  • Cv2
  • Glob
  • Shutil
  • PyOctree

Data Convention

Organize your data in the ./data/ folder following this structure:

./data/FOLDER/
    albedo/          # (Optional)
        000.png
        001.png
        002.png
    normal/          # (Mandatory)
        000.png
        001.png
        002.png
    mask/            # (Mandatory)
        000.png
        001.png
        002.png
    mask_normal_uncertainty/  # (Optional)
        000.png
        001.png
        002.png
    cameras.npz

Data

We provide the DiLiGenT-MV dataset with normals and reflectance maps estimated using SDM-UniPS. Reflectance maps were scaled over all views, and uncertainty masks were generated from 100 normal estimations (see the paper for details).

Training

Preprocess the Data

python script/preprocess.py --folder ./data/<FOLDER>/ --exp_name <EXP_NAME>

Run Optimization

./run.sh ./data/<FOLDER>/<EXP_NAME>

Results will be stored in ./data/<FOLDER>/<EXP_NAME>/. Modify the ./build/testbed command in the run.sh with the following options:

--scene FOLDER          # Path to your data
--maxiter INT           # Number of iterations
--mask-weight FLOAT     # Weight of the mask loss
--save-mesh             # Extract the mesh at the end
--save-snapshot         # Save the neural weights
--no-albedo             # Train only on normals
--resolution INT        # Resolution for marching cube (default 512)
--no-gui                # Run optimization without GUI

Run Optimization with Scaled Reflectance Maps

For reflectance maps with varying scale factors, use the --scale-albedo flag that generates a mesh without reflectance maps first, then uses this mesh to scale the reflectance maps (pyoctree and scipy are needed in a python environment). Finally, it generates a mesh using the scaled reflectance maps.

./run.sh ./data/<FOLDER>/<EXP_NAME> --scale-albedo

Results will be stored in ./data/<FOLDER>/<EXP_NAME>-albedoscaled/.

Note: The provided DiLiGenT-MV dataset already has scaled reflectance maps in the albedo folder.

Acknowledgements & Citation

@inproceedings{Brument24,
    title={RNb-NeuS: Reflectance and Normal-based Multi-View 3D Reconstruction},
    author={Baptiste Brument and Robin Bruneau and Yvain Quéau and Jean Mélou and François Lauze and Jean-Denis Durou and Lilian Calvet},
    booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2024}
}
@inproceedings{neus2,
    title={NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction}, 
    author={Wang, Yiming and Han, Qin and Habermann, Marc and Daniilidis, Kostas and Theobalt, Christian and Liu, Lingjie},
    year={2023},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}
}
@article{mueller2022instant,
    author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
    title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
    journal = {ACM Trans. Graph.},
    issue_date = {July 2022},
    volume = {41},
    number = {4},
    month = jul,
    year = {2022},
    pages = {102:1--102:15},
    articleno = {102},
    numpages = {15},
    url = {https://doi.org/10.1145/3528223.3530127},
    doi = {10.1145/3528223.3530127},
    publisher = {ACM},
    address = {New York, NY, USA},
}
@inproceedings{wang2021neus,
    title={NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction},
    author={Wang, Peng and Liu, Lingjie and Liu, Yuan and Theobalt, Christian and Komura, Taku and Wang, Wenping},
    booktitle={Proc. Advances in Neural Information Processing Systems (NeurIPS)},
    volume={34},
    pages={27171--27183},
    year={2021}
}