Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MM] Add support for probing and loading SDXL VAE checkpoint files #6524

Merged
merged 2 commits into from
Jun 20, 2024

Conversation

lstein
Copy link
Collaborator

@lstein lstein commented Jun 20, 2024

Summary

This adds support for probing and converting SDXL VAE checkpoint files. It is a lame solution because the base-determining method simply looks for the pattern "SDXL" in the filename. A quick inspection of SD1 vs SDXL VAEs found no differences in the state dict weights, so these can't be used as the basis of discrimination.

Conversion seems to work using the diffusers library. However, the need to convert will be obsoleted by PR #6510 .

Related Issues / Discussions

See Discord thread at https://discord.com/channels/1020123559063990373/1020123559831539744/1253157322377793557

QA Instructions

I've tested on the Civitai VAE https://civitai.com/models/140686/fix-fp16-errors-sdxl-lower-memory-use-sdxl-vae-fp16-fix-by-madebyollin. The results aren't great - the colors are washed out - but I don't know if this is a problem with the VAE or with the conversion.

Merge Plan

Merge when approved.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@github-actions github-actions bot added python PRs that change python files backend PRs that change backend files labels Jun 20, 2024
@hipsterusername
Copy link
Member

Should we just punt on this? Especially since it seems like it's going to be unreliable - I can only think of one VAE named with "SDXL" in it

@lstein
Copy link
Collaborator Author

lstein commented Jun 20, 2024

Should we just punt on this? Especially since it seems like it's going to be unreliable - I can only think of one VAE named with "SDXL" in it

I did a quick search for all SDXL VAEs on Civitae, and each one has the string "XL" in the name. None of the SD 1.5 models contain this string. I changed the regular expression to look for "XL", and probing will work for now. If some future model has its base type incorrectly called the user can always edit it.

I'm going to go ahead and merge in. This is a rare use case anyway.

@lstein lstein enabled auto-merge (squash) June 20, 2024 02:53
@lstein lstein merged commit b03073d into main Jun 20, 2024
14 checks passed
@lstein lstein deleted the lstein/bugfix/probe-sdxl-vae-checkpoints branch June 20, 2024 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files python PRs that change python files
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants