Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Community Pipeline] Add some feature for regional prompting pipeline #9874

Merged
merged 12 commits into from
Nov 28, 2024

Conversation

cjkangme
Copy link
Contributor

@cjkangme cjkangme commented Nov 6, 2024

What does this PR do?

Hi.
I am happily using a pipeline that implements Regional Prompting in the diffusers community pipeline.
While using the community pipeline, I created a PR as I thought it would be good to add the following:

  • Fix negative prompt processing logic (minor error)
  • Add Use base prompt logic
  • Fix initialization errors occurring in the latest version

Using Example

I tried to imitate this example

2d animation style ADDBASE
masterpiece, high quality ADDCOMM
blue sky BREAK
green hair twintail BREAK
book shelf BREAK
messy desk BREAK
orange++ dress and sofa
mode: rows
div: 1,2,1,1;2,4,6
base_ratio: 0.3

output

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul
Copy link
Member

Thanks for your PR! Do you maybe also want to update the README of regional prompting?

@cjkangme
Copy link
Contributor Author

cjkangme commented Nov 8, 2024

@sayakpaul
sure, I have made updates regarding the use_base option

@@ -3406,6 +3420,7 @@ pipe(prompt=prompt, rp_args=rp_args)
### Optional Parameters

- `save_mask`: In `Prompt` mode, choose whether to output the generated mask along with the image. The default is `False`.
- `base_ratio`: Used with `ADDBASE`. Sets the ratio of the base prompt; if base ratio is set to 0.2, then resulting images will consist of `20%*BASE_PROMPT + 80%*REGION_PROMPT`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could maybe also specify the reference that introduced this technique?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This technique was originally implemented in hako-mikan's regional-prompter, but not in the current community pipeline.
So, I don't think it's necessary to specify the reference again.

@sayakpaul
Copy link
Member

@cjkangme could you run make style && make quality and push your changes?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@cjkangme
Copy link
Contributor Author

cjkangme commented Nov 8, 2024

@sayakpaul
I accidentally ran make style in the root directory and it modified some wrong files. So I reverted it 😅😅
nothing changed about the regional prompting

@cjkangme
Copy link
Contributor Author

@sayakpaul

Hi
I've reapplied the make style, can you confirm?

Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@sayakpaul sayakpaul merged commit 69c83d6 into huggingface:main Nov 28, 2024
8 checks passed
@vladmandic
Copy link
Contributor

was this pr tested? i'm getting runtime error since the merge:

│ /home/vlado/.cache/huggingface/modules/diffusers_modules/git/regional_prompting_stable_diffusion.py:462 in promptsmaker                                                                                                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                                                                                                                                                                                  │
│   461 │   │   │   add = add.strip() + " "                                                                                                                                                                                                                                                                                                                                                                                        │
│ ❱ 462 │   │   prompts = [p.strip() for p in prompt.split(KBRK)]                                                                                                                                                                                                                                                                                                                                                                  │
│   463 │   │   out_p.append([add + p for i, p in enumerate(prompts)])                                                                                                                                                                                                                                                                                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'list' object has no attribute 'split'

@cjkangme
Copy link
Contributor Author

@vladmandic

Hi
I apologize for the issue.
There was a typo in my code that caused an error when prompts is a str type and negative_prompts is a list type

I've fixed the affected line and created PR #10251

sayakpaul added a commit that referenced this pull request Dec 23, 2024
…#9874)

* [Fix] fix bugs of  regional_prompting pipeline

* [Feat] add base prompt feature

* [Fix] fix __init__ pipeline error

* [Fix] delete unused args

* [Fix] improve string handling

* [Docs] docs to use_base in regional_prompting

* make style

---------

Co-authored-by: Sayak Paul <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants