Skip to content

Latest commit

 

History

History
87 lines (64 loc) · 4.46 KB

README.md

File metadata and controls

87 lines (64 loc) · 4.46 KB

Get started with StreamV2V

English | 日本語

Prepartion

We recommend to use gdown to prepare data and models.

# Install gdown
pip install gdown
pip install --upgrade gdown

Download evaluation videos.

cd vid2vid
gdown https://drive.google.com/drive/folders/1q963FU9I4I8ml9_SeaW4jLb4kY3VkNak -O demo_selfie --folder

(Recommended) Download lora weights for better stylization.

# Make sure you are under the directory of vid2vid
gdown https://drive.google.com/drive/folders/1D7g-dnCQnjjogTPX-B3fttgdrp9nKeKw -O lora_weights --folder
Trigger words LORA weights Source
'pixelart' , 'pixel art' , 'Pixel art' , 'PixArFK' Google drive Civitai
'lowpoly', 'low poly', 'Low poly' Google drive Civitai
'Claymation', 'claymation' Google drive Civitai
'crayons', 'Crayons', 'crayons doodle', 'Crayons doodle' Google drive Civitai
'sketch', 'Sketch', 'pencil drawing', 'Pencil drawing' Google drive Civitai
'oil painting', 'Oil painting' Google drive Civitai

Evaluation

# Evaluate a single video
python main.py --input ./demo_selfie/jeff_1.mp4 --prompt "Elon Musk is giving a talk."
python main.py --input ./demo_selfie/jeff_1.mp4 --prompt "Claymation, a man is giving a talk."
# Evaluate a batch of videos
python batch_eval.py --json_file ./demo_selfie/eval_jeff_celebrity.json # Face swap edits
python batch_eval.py --json_file ./demo_selfie/eval_jeff_lorastyle.json # Stylization edits

CAUTION: The --acceleration tensorrt option is NOT SUPPORTED! I did try to accelerate the model with TensorRT, but due to the dynamic nature of the feature bank, I didn't succeed. If you are an expert on this, please contact me ([email protected]) and we could discuss how to include you as a contributor.

Of course, you can also use the front-end page provided by Gradio. You need to execute pip install gradio to install gradio and start it using python main_gr.py

Ablation study using command

# Do not use feature bank, the model would roll back into per-frame StreamDiffusion
python main.py --input ./demo_selfie/jeff_1.mp4 --prompt "Claymation, a man is giving a talk." --use_cached_attn False --output_dir outputs_streamdiffusion
# Specify the noise strength. Higher the noise_strength means more noise is added to the starting frames.
# Highter strength ususally leads to better edit effects but may sacrifice the consistency. By default, it is 0.4.
python main.py --input ./demo_selfie/jeff_1.mp4 --prompt "Claymation, a man is giving a talk." --noise_strength 0.8 --output_dir outputs_strength
# Specify the diffusion steps. Higher steps ususally lead to higher quality but slower speed.
# By default, it is 4.
python main.py --input ./demo_selfie/jeff_1.mp4 --prompt "Claymation, a man is giving a talk." --diffusion_steps 1 --output_dir outputs_steps

Common Bugs

ImportError Issue

  • Error Message: ImportError: cannot import name 'packaging' from 'pkg_resources'.
  • Related GitHub Issue: setuptools issue #4961

Potential Workaround:
Downgrade the setuptools package to resolve this issue. You can do this by running the following command in your terminal:

pip install setuptools==69.5.1