Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simplify docs part ii #190

Merged
merged 5 commits into from
Jan 7, 2025
Merged

simplify docs part ii #190

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0ad57ed
simplify docs.
sayakpaul Jan 7, 2025
a83a626
clarify the cog checkpoints supported.
sayakpaul Jan 7, 2025
c45ac22
make the note smaller.
sayakpaul Jan 7, 2025
d14f540
replace with sub
sayakpaul Jan 7, 2025
6fe986a
Apply suggestions from code review
sayakpaul Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,14 +135,16 @@ For inference, refer [here](./docs/training/ltx_video.md#inference). For docs re

<div align="center">

| Model Name | Tasks | Ckpts Tested | Min. GPU<br>VRAM | Comments |
|:------------:|:---------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------:|:----------------------:|:--------------------------------------------------------------:|
| [LTX-Video](https://huggingface.co/docs/diffusers/main/api/pipelines/ltx_video) | <ul><li>T2V ✅</li><li> I2V ❌</li></ul> | [Lightricks/LTX-Video](https://huggingface.co/Lightricks/LTX-Video) | 11 GB | Fast to train |
| [HunyuanVideo](https://huggingface.co/docs/diffusers/main/api/pipelines/hunyuan_video) | <ul><li>T2V ✅</li><li> I2V ❌</li></ul> | [tencent/HunyuanVideo](https://huggingface.co/tencent/HunyuanVideo) | 42 GB | - |
| [CogVideoX](https://huggingface.co/docs/diffusers/main/api/pipelines/cogvideox) | <ul><li>T2V ✅</li><li> I2V ❌</li></ul> | <ul><li>[THUDM/CogVideoX1.5-5B](https://huggingface.co/THUDM/CogVideoX1.5-5B)</li><li>[THUDM/CogVideoX-5b](https://huggingface.co/THUDM/CogVideoX-5b)</li><li>[THUDM/CogVideoX-2b](https://huggingface.co/THUDM/CogVideoX-2b)</li> | - GB | Training with multi-bucket, multi-resolution frames is supported. |
| **Model Name** | **Tasks** | **Min. GPU VRAM** |
|:---:|:---:|:---:|
| [LTX-Video](./docs/training/ltx_video.md) | Text-to-Video | 11 GB |
| [HunyuanVideo](./docs/training/hunyuan_video.md) | Text-to-Video | 42 GB |
| [CogVideoX](./docs/training/cogvideox.md) | Text-to-Video | 12GB<sup>*</sup> |

</div>

<sub><sup>*</sup>Noted for the 5B variant.</sub>

Note that the memory consumption in the table is reported with most of the options, discussed in [docs/training/optimizations](./docs/training/optimization.md), enabled.

If you would like to use a custom dataset, refer to the dataset preparation guide [here](./docs/dataset/README.md).
Expand Down
11 changes: 10 additions & 1 deletion docs/training/cogvideox.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,14 @@ Training configuration: {
| after validation end | 11.145 | 28.324 |
| after training end | 11.144 | 11.592 |

## Supported checkpoints

CogVideoX has multiple checkpoints as one can note [here](https://huggingface.co/collections/THUDM/cogvideo-66c08e62f1685a3ade464cce). The following checkpoints were tested with `finetrainers` and are known to be working:

* [THUDM/CogVideoX-2b](https://huggingface.co/THUDM/CogVideoX-2b)
* [THUDM/CogVideoX1.5-5B](https://huggingface.co/THUDM/CogVideoX1.5-5B)
sayakpaul marked this conversation as resolved.
Show resolved Hide resolved
* [THUDM/CogVideoX1.5-5B](https://huggingface.co/THUDM/CogVideoX1.5-5B)

## Inference

Assuming your LoRA is saved and pushed to the HF Hub, and named `my-awesome-name/my-awesome-lora`, we can now use the finetuned model for inference:
Expand All @@ -128,7 +136,8 @@ video = pipe("<my-awesome-prompt>").frames[0]
export_to_video(video, "output.mp4")
```

You can refer to the following guides to know more about performing LoRA inference in `diffusers`:
You can refer to the following guides to know more about the model pipeline and performing LoRA inference in `diffusers`:

* [CogVideoX in Diffusers](https://huggingface.co/docs/diffusers/main/en/api/pipelines/cogvideox)
* [Load LoRAs for inference](https://huggingface.co/docs/diffusers/main/en/tutorials/using_peft_for_inference)
* [Merge LoRAs](https://huggingface.co/docs/diffusers/main/en/using-diffusers/merge_loras)
3 changes: 2 additions & 1 deletion docs/training/hunyuan_video.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,8 @@ output = pipe(
export_to_video(output, "output.mp4", fps=15)
```

You can refer to the following guides to know more about performing LoRA inference in `diffusers`:
You can refer to the following guides to know more about the model pipeline and performing LoRA inference in `diffusers`:

* [LTX-Video in Diffusers](https://huggingface.co/docs/diffusers/main/api/pipelines/hunyuan_video)
sayakpaul marked this conversation as resolved.
Show resolved Hide resolved
* [Load LoRAs for inference](https://huggingface.co/docs/diffusers/main/en/tutorials/using_peft_for_inference)
* [Merge LoRAs](https://huggingface.co/docs/diffusers/main/en/using-diffusers/merge_loras)
3 changes: 2 additions & 1 deletion docs/training/ltx_video.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,8 @@ video = pipe("<my-awesome-prompt>").frames[0]
export_to_video(video, "output.mp4", fps=8)
```

You can refer to the following guides to know more about performing LoRA inference in `diffusers`:
You can refer to the following guides to know more about the model pipeline and performing LoRA inference in `diffusers`:

* [LTX-Video in Diffusers](https://huggingface.co/docs/diffusers/main/en/api/pipelines/ltx_video)
* [Load LoRAs for inference](https://huggingface.co/docs/diffusers/main/en/tutorials/using_peft_for_inference)
* [Merge LoRAs](https://huggingface.co/docs/diffusers/main/en/using-diffusers/merge_loras)