A Light Neural Network To Control Stable Diffusion Spatial Information tuned by Chinese
ControlLoRA is a project that can easily fine-tune stable diffusion to achieve the purpose of controlling its spatial information, with LoRA, a simple and small (~7M parameters, ~25M storage space) network. More information can be seen in ControlLoRA.
This project can be seen as a fork from ControlLoRA and give two models in Chinese domain use ControlLoRA's source code.
You can run them in the huggingface space online, Upload your own image and input Chinese prompt. But because the deployments are on cpu device. I recommend you download them to local and run them in your gpu. (they will switch the device dynamical because "is_available")
| Name | HuggingFace Model link | HuggingFace Space link |
|---|---|---|
| ControlNet By Canny Chinese 🔪 | https://huggingface.co/svjack/canny-control-lora-zh | https://huggingface.co/spaces/svjack/ControlNet-Canny-Chinese |
| ControlNet By Pose Chinese 🏃 | https://huggingface.co/svjack/pose-control-lora-zh | https://huggingface.co/spaces/svjack/ControlNet-Pose-Chinese |
pip install -r requirements.txtAfter installtion, One can cd into ControlNet-Canny-Chinese and ControlNet-Pose-Chinese and run
python app.pyrespectively
Open your browser ,go to http://localhost:7860 to try it on browser.
| Name | Prompt | Original Image | Backbone Image | Transformed Image |
|---|---|---|---|---|
| ControlNet By Canny Chinese 🔪 | 表情俏皮的小丑 |  |
 |
 |
| ControlNet By Canny Chinese 🔪 | 满布流星的夜晚 |  |
 |
 |
| ControlNet By Canny Chinese 🔪 | 猫咪吸血鬼 |  |
 |
 |
| ControlNet By Pose Chinese 🏃 | 麦田守望者 |  |
 |
 |
| ControlNet By Pose Chinese 🏃 | 身穿军服的军官 |  |
 |
 |
LoRA: Low-Rank Adaptation of Large Language Models
LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights. This vastly reduces the storage requirement for large language models adapted to specific tasks and enables efficient task-switching during deployment all without introducing inference latency. LoRA also outperforms several other adaptation methods including adapter, prefix-tuning, and fine-tuning.
In the Stable Diffusion domain, I also provide three Stable Diffusion model finetuned with the help of Lora.
The CC3M dataset download and transformed with the help of repo svjack/img2dataset-pq2hf-transform-toolkit
| Name | HuggingFace Model link | Language | Tuned Dataset |
|---|---|---|---|
| svjack/pokemon-sd-lora-zh | https://huggingface.co/svjack/pokemon-sd-lora-zh | Chinese | svjack/pokemon-blip-captions-en-zh |
| svjack/concept-caption-3m-sd-lora-en | https://huggingface.co/svjack/concept-caption-3m-sd-lora-en | English | Conceptual Captions (CC3M) |
| svjack/concept-caption-3m-sd-lora-zh | https://huggingface.co/svjack/concept-caption-3m-sd-lora-zh | Chinese | Conceptual Captions (CC3M) |
You can check the model card to understand how to use them.
svjack - [email protected] - [email protected]
Project Link:https://github.com/svjack/ControlLoRA-Chinese