[Bug]: crash when using sdxl loras

[TongfanWeitf]

Checklist

• The issue exists after disabling all extensions
• The issue exists on a clean installation of webui
• The issue is caused by an extension, but I believe it is caused by a bug in the webui
• The issue exists in the current version of the webui
• The issue has not been reported before recently
• The issue has been reported before but has not been fixed yet

What happened?

if i use sdxl loras, webui will crash.

Steps to reproduce the problem

1.run webui.
2.run a txt2img with sdxl model and lora
3.crash

What should have happened?

successfully return the img

What browsers do you use to access the UI ?

Microsoft Edge

Sysinfo

sysinfo-2024-03-07-18-39.json

Console logs

Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.8.0
Commit hash: bef51aed032c0aaa5cfd80445bc4cf0d85b408b5
Launching Web UI with arguments: --xformers --no-half-vae --no-half --medvram-sdxl
Loading weights [67ab2fd8ec] from D:\ai\webui\models\Stable-diffusion\ponyDiffusionV6XL_v6StartWithThisOne.safetensors
Creating model from config: D:\ai\webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 16.4s (prepare environment: 3.4s, import torch: 5.9s, import gradio: 0.6s, setup paths: 0.8s, initialize shared: 3.2s, other imports: 0.6s, load scripts: 0.8s, create ui: 0.5s, gradio launch: 0.6s).
Loading VAE weights specified in settings: D:\ai\webui\models\VAE\sdxl_vae.safetensors
Applying attention optimization: xformers... done.
Model loaded in 20.6s (load weights from disk: 0.7s, create model: 1.9s, apply weights to model: 7.4s, apply float(): 4.6s, load VAE: 0.7s, calculate empty prompt: 5.3s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:25<00:00,  1.28s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:27<00:00,  1.40s/it]
  0%|                                                                                           | 0/20 [00:00<?, ?it/s] 请按任意键继续. . .
//the first bar is without lora and the second one is with lora. it crashed so no error messages. the chinese at the end means "press any key to continue..."

Additional information

it is weird, beacuse I can run sdxl with loras before.
In some day I suddently cant load sdxl models (pytorch allocated 10.6G which is much more than before), so I add --medvram-sdxl. Now I can load sdxl models, but I still cant use loras.

[TongfanWeitf]

i uploaded my gpu drive, now it reaches the end of progress:
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.8.0
Commit hash: bef51ae
Launching Web UI with arguments: --xformers --no-half-vae --no-half --medvram-sdxl
Loading weights [67ab2fd8ec] from D:\ai\webui\models\Stable-diffusion\ponyDiffusionV6XL_v6StartWithThisOne.safetensors
Creating model from config: D:\ai\webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
Startup time: 13.2s (prepare environment: 3.1s, import torch: 6.0s, import gradio: 0.6s, setup paths: 0.8s, initialize shared: 0.2s, other imports: 0.5s, load scripts: 0.8s, create ui: 0.5s, gradio launch: 0.5s).
Applying attention optimization: xformers... done.
Model loaded in 16.6s (load weights from disk: 0.7s, create model: 2.2s, apply weights to model: 7.4s, apply float(): 4.9s, calculate empty prompt: 1.3s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:51<00:00, 2.57s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:33<00:00, 1.75s/it]

but will still crash after that(means I cant receive the img, though i can see it is gerenated, but it crashed the moment it finished, the eta in webui is like 90%/3s left)

[nailz420]

Can you check in windows event logs for any related messages? Python crash details? Resource exhaustion?

[TongfanWeitf]

Can you check in windows event logs for any related messages? Python crash details? Resource exhaustion?

i already solved it. when I want to load sdxl models, I use --medvram-sdxl, then I restart without --medvram-sdxl and then I can used sdxl models. If I want to load another model, I restart with --medvram-sdxl again and do such thing again.

[nailz420]

Can you check in windows event logs for any related messages? Python crash details? Resource exhaustion?

i already solved it. when I want to load sdxl models, I use --medvram-sdxl, then I restart without --medvram-sdxl and then I can used sdxl models. If I want to load another model, I restart with --medvram-sdxl again and do such thing again.

That doesn't sound like a solution, but a clunky workaround. The issue still exists if you have to use that

[aurelion314]

I have the same issue, but the solution of using --medvram-sdxl then restarting does not solve it for me.

The sdxl model works fine on its own, but as soon as I add the lora it crashes. Typically as soon as I try to generate, my computer temporarily locks up, then the ui crashes with this in the console:

0%| | 0/8 [00:00<?, ?it/s]./webui.sh: line 292: 11929 Killed "${python_cmd}" -u "${LAUNCH_SCRIPT}" "$@"

Running Linux Mint. Using --no-half --precision full

Edit:
I have discovered that when generating with a lora, it consumes all available RAM and then crashes. I suspect it is loading the entire base model into memory again, creating a copy and doubling memory usage. When only the base model is loaded, RAM usage is less than 20gb of 32gb. When I start the generation with the lora, roughly 3gb is added every seconds until it hits the full 32gb, which is when it either crashes or locks the PC.

I confirmed this by using SD 1.5, which is small enough that I can have 2 full copies in memory. Generating with a lora again starts by consuming a bunch of RAM, but stops at about 6gb of additional memory (the lora itself is only 150mb), then everything works and the image generates just fine.

I'm guessing that isn't supposed to happen? Shouldn't it either use the model already in memory, or free up that space if it is going to reload the whole thing?

[aearone]

I have the same issue, but the solution of using --medvram-sdxl then restarting does not solve it for me.

The sdxl model works fine on its own, but as soon as I add the lora it crashes. Typically as soon as I try to generate, my computer temporarily locks up, then the ui crashes with this in the console:

0%| | 0/8 [00:00<?, ?it/s]./webui.sh: line 292: 11929 Killed "${python_cmd}" -u "${LAUNCH_SCRIPT}" "$@"

Running Linux Mint. Using --no-half --precision full

Edit: I have discovered that when generating with a lora, it consumes all available RAM and then crashes. I suspect it is loading the entire base model into memory again, creating a copy and doubling memory usage. When only the base model is loaded, RAM usage is less than 20gb of 32gb. When I start the generation with the lora, roughly 3gb is added every seconds until it hits the full 32gb, which is when it either crashes or locks the PC.

I confirmed this by using SD 1.5, which is small enough that I can have 2 full copies in memory. Generating with a lora again starts by consuming a bunch of RAM, but stops at about 6gb of additional memory (the lora itself is only 150mb), then everything works and the image generates just fine.

I'm guessing that isn't supposed to happen? Shouldn't it either use the model already in memory, or free up that space if it is going to reload the whole thing?

I can confirm that when using the SDXL model (in my case PonyDiffusion), the RAM consumption increases with every second of generation until it reaches 15.6 GB (I have 16 GB) and the swap file starts to be used. That said, if I use Lora, the RAM gets clogged up after the first generation and it says "Press any button...". I perfectly remember using SDXL model 2 months ago without any problems, I don't remember such RAM consumption. And the error is not on the video memory side of the GPU. Has anyone found a solution?

[aearone]

I figured out what the problem is. The issue is a RAM leak in the latest version of webui, i.e. during generation the RAM usage grows every second until it reaches the swap file limit. In webui version 1.7.0 there was no such thing. It's just that RAM is at its maximum, it is enough for the first/second generation. But when using Lora, it runs out almost instantly. Apparently there is a bug somewhere that the model is unloaded into RAM at every generation. After downgrading the webui version, the memory leaks no longer occur and the images are generated normally:)
As I understand it, this is because of PyTorch 2.1.2, which was added to webui 1.8.0 version.

Let's wait for fixes, if there are any in the future.

You can download version 1.7.0 manually here:: https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/v1.7.0

@lllyasviel/stable-diffusion-webui-forge#500 (comment) @#15206 (comment)

[Les-Tin]

#15348

[Stellaris12]

I figured out what the problem is. The issue is a RAM leak in the latest version of webui, i.e. during generation the RAM usage grows every second until it reaches the swap file limit. In webui version 1.7.0 there was no such thing. It's just that RAM is at its maximum, it is enough for the first/second generation. But when using Lora, it runs out almost instantly. Apparently there is a bug somewhere that the model is unloaded into RAM at every generation. After downgrading the webui version, the memory leaks no longer occur and the images are generated normally:) As I understand it, this is because of PyTorch 2.1.2, which was added to webui 1.8.0 version.

Let's wait for fixes, if there are any in the future.

You can download version 1.7.0 manually here:: https://github.com/AUTOMATIC1111/stable-diffusion-webui/tree/v1.7.0

@lllyasviel/stable-diffusion-webui-forge#500 (comment) @#15206 (comment)

I don't think torch 2.1.2 is the only cause of the issue as I am having this same issue with my ARC gpu which uses torch: 2.0.0a0+gite9ebda2 in version 1.8 of the webui.

[TongfanWeitf]

Can you check in windows event logs for any related messages? Python crash details? Resource exhaustion?

i already solved it. when I want to load sdxl models, I use --medvram-sdxl, then I restart without --medvram-sdxl and then I can used sdxl models. If I want to load another model, I restart with --medvram-sdxl again and do such thing again.

That doesn't sound like a solution, but a clunky workaround. The issue still exists if you have to use that

sry I cant give any debug logs. I am using google cloud and thus I can get rid of this bug. Just by restart the vm.

[bandifiu]

I had similar problem with RAM being filled after each generation when I tried to compare models with x/y/z plot. With v1.8 increasing the number of model loaded into RAM crashing the system after few generations. Memory is allocated after each generation even when model is used from cache. For me using the old settings (obsolate one) for increasing the number of cached models instead the new one fixed the crashes.

[silveroxides]

This might be similar to my issues so I will post here instead of making new issue.

Console Log

Events look like this

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: python.exe (52724) consumed 21200195584 bytes, msedge.exe (46448) consumed 6455615488 bytes, and python.exe (10724) consumed 6229950464 bytes.

Bytes converted.
It shouold not consume that much right?

Event Viewer 1

Event Viewer 2

[silveroxides]

@w-e-w You might want to look at this
Update:
I can somewhat see what might be happening now.
Several times now when decoding image with VAE it leaves shared memory occupied while it drops some weights from the loaded checkpoint it seems which it then has to load back up when generation starts.

1. End of normal run with decode start
2. Decode where shared memory is used
3. Decode finishes
4. Normal loaded model weights
5. Changing model

1. End of normal run with decode start
2. Decode where shared memory is used
3. Decode finishes and something is left in shared while VRAM is below normal
4. Initiating next generation
5. Actually starting to generate image

1. Normal generation ongoing
2. Decode start
3. Decore using shared memory
4. Same as previous with odd memory usage
5. Sketchy workaround with opening settings and unloading model to RAM and then back to VRAM to get it back to normal

[God-damnit-all]

I notice that the 1.9 release candidate made some changes regarding loras, has anyone tested to see if it's fixed on there?

[Takezo1000]

I tested SDXL loras with 1.9 and RAM usage still skyrockets to the maximum (32 GB) and then it starts using virtual memory (using up to 20 GB of virtual memory). Only happens when using SDXL Loras.

[God-damnit-all]

I tested SDXL loras with 1.9 and RAM usage still skyrockets to the maximum (32 GB) and then it starts using virtual memory (using up to 20 GB of virtual memory). Only happens when using SDXL Loras.

Is this with the now-released 1.9?

[Takezo1000]

I tested SDXL loras with 1.9 and RAM usage still skyrockets to the maximum (32 GB) and then it starts using virtual memory (using up to 20 GB of virtual memory). Only happens when using SDXL Loras.

Is this with the now-released 1.9?

Yes, with 1.9 it still has this problem. I'm using --medvram --medvram-sdxl arguments with RX 6700 XT (AMD)

[God-damnit-all]

I tested SDXL loras with 1.9 and RAM usage still skyrockets to the maximum (32 GB) and then it starts using virtual memory (using up to 20 GB of virtual memory). Only happens when using SDXL Loras.

Is this with the now-released 1.9?

Yes, with 1.9 it still has this problem. I'm using --medvram --medvram-sdxl arguments with RX 6700 XT (AMD)

That's unfortunate. I'd downgrade to 1.7 but that has problems of its own. Could it be related to xformers? Try setting Cross attention optimization to None in Optimizations.

[silveroxides]

I tested SDXL loras with 1.9 and RAM usage still skyrockets to the maximum (32 GB) and then it starts using virtual memory (using up to 20 GB of virtual memory). Only happens when using SDXL Loras.

Is this with the now-released 1.9?

Yes, with 1.9 it still has this problem. I'm using --medvram --medvram-sdxl arguments with RX 6700 XT (AMD)

That's unfortunate. I'd downgrade to 1.7 but that has problems of its own. Could it be related to xformers? Try setting Cross attention optimization to None in Optimizations.

The following is what I did and it worked wonders. I get the feeling that the last two releases implemented a lot of arbritrary changes which had no justifiable reason to be implemented to a main branch. This include spandrel which has cause major slowdowns for a big part of users(yes you might not see them report here cause non tech savvy users get put off being requested to spend an hour learning how to properly submit an issue). I suspect they missed reading the full changelog for Torch releases 2.1.0-2.1.2 in regards to weights and multiple devices (in this case their odd loading between cpu and gpu)
Anyway I hope this helps

Instructions:

Downgrade to 1.8.0

Install CUDA toolkit 11.8

Install cudnn

"extract respective files into CUDA installs bin,include and lib\x64 folders"

Click windows search bar

and type environment then click "Edit system environment variables" in results and in next window that pops up.

Editing variables

In the lower part check that you have CUDA_PATH set to same as CUDA_PATH_V11_8 and add one named CUDNN and give it the variable(assuming you install instandard location. edit the following if different)

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\lib\x64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include;

After all this is done

Reboot your PC once.

Open your webui root folder

Open Powershell and acitvate the venv

.\venv\Scrips\activate

Then install torch 2.0.1 for cuda11.8

pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118

Install xformers 0.0.22

pip install xformers==0.0.22

Lastly do check just in case there is something important that conflicts but there rarely is.

pip check

Open webui-user.bat in text editor and add --skip-install just in case.

Hope this works. Will try to check back later to see if all went well

[JeffreyLebino]

Hi, happened to me this weekend as well,

Tried rolling back the 1.9.3 back to 1.9 then 1.8 as well as deleting the venv before coming here but it seems it didnt work.
Anytime i try to gen on 1.5 the ram goes up then returns to about a fourth of the total available.
With SDXL it seems there is a memory leak and it climbs with each gen, sometimes lowering a little but eventually filling the total.

Tried on 16 and 32gb of ram and i get the same effect on both.
Adding more loras make the server crash earlier.
I tried switching to 1.5 models after each XL gens but it seems to be mostly placebo.
tried switching off xformers but no results either.

It pretty much always ends with an error message or with just "press any key to continue..."

[aearone]

Hi, happened to me this weekend as well,

Tried rolling back the 1.9.3 back to 1.9 then 1.8 as well as deleting the venv before coming here but it seems it didnt work. Anytime i try to gen on 1.5 the ram goes up then returns to about a fourth of the total available. With SDXL it seems there is a memory leak and it climbs with each gen, sometimes lowering a little but eventually filling the total.

Tried on 16 and 32gb of ram and i get the same effect on both. Adding more loras make the server crash earlier. I tried switching to 1.5 models after each XL gens but it seems to be mostly placebo. tried switching off xformers but no results either.

It pretty much always ends with an error message or with just "press any key to continue..."

I can attest to that. The same thing is happening to me. Same problem on version 1.8.0, the failures are less but after a few generations there is still a RAM error. I found out that the problem lies in the command --medvram, it frees the video memory and very heavily loads the RAM, while the VRAM barely uses 6.5 gb (out of 8 in my case). If you remove this command, the generation takes a very long time. I don't know what's broken in the latest webui updates, but it's a fact. There is a leak of RAM when it is used at 100%, and if you use LoRA it happens even faster.

[nailz420]

Noticed that this memory leak is not triggered during X/Y/Z script runs

[JeffreyLebino]

Ok i may have a workaround that limits the amount of crashes, while using Kohya i had the same issue an it resolved both,
mind you it still fills your ram to an absurd amount (31.9/32 for me)
it seems to just avoid the final crash when it tries to add more to it

Disabling system memory fallback helps by stopping the gpu from dumping the excess of VRAM into RAM (it might even speed up your 1.5 gens if you havent done it in the past)
Theres still something filling 99.9% of the system RAM, and SD still redlines all the time but at least the gpu doesnt dump more into that 0.1%

https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion
Just be sure to do it on the right python, the venv or the main system one depending on which one your SD uses.

[nailz420]

Noticed that this memory leak is not triggered during X/Y/Z script runs

Running X/Y/Z plot script actually clears out the VM

[nailz420]

Ok i may have a workaround that limits the amount of crashes, while using Kohya i had the same issue an it resolved both, mind you it still fills your ram to an absurd amount (31.9/32 for me) it seems to just avoid the final crash when it tries to add more to it

Disabling system memory fallback helps by stopping the gpu from dumping the excess of VRAM into RAM (it might even speed up your 1.5 gens if you havent done it in the past) Theres still something filling 99.9% of the system RAM, and SD still redlines all the time but at least the gpu doesnt dump more into that 0.1%

https://nvidia.custhelp.com/app/answers/detail/a_id/5490/~/system-memory-fallback-for-stable-diffusion Just be sure to do it on the right python, the venv or the main system one depending on which one your SD uses.

You're supposed to do that for SD regardless of this bug, especially with 8gb VRAM or less

[JeffreyLebino]

You're supposed to do that for SD regardless of this bug, especially with 8gb VRAM or less

Sure but OP and i both had crashes as well as the memory leak hence why i shared it, two problems in one sort of things. I experienced way less issues with the fix.
And SDXL was working fine without it on my side so far.

[AlanMW]

My webUI stopped hanging up when I removed the --medvram flag as well. It cuts my speed from 6.5 it/s to 3 it/s but at least it runs.

[nailz420]

I use Intelligent standby list cleaner (ISLC). 8gb VRAM, 32gb RAM. I can generate for about 20-30 mins before crashing with medvram-sdxl and xformers.

[aearone]

I use Intelligent standby list cleaner (ISLC). 8gb VRAM, 32gb RAM. I can generate for about 20-30 mins before crashing with medvram-sdxl and xformers.

So it's a massive problem. We should do something about it and draw the contributors attention to it.

[PKK02]

I've been facing the same exact problem for weeks now. I was able to use XL loras before with no problems and now it crashes my entire system even if I use it only once. It will go through the first gen but if I try to generate other images, it slowly freezes my computer even when not using loras. Very annoying. I'm using a GTX1080 on Arch Linux. Latest drivers, latest webui, --medvram-sdxl flag.

[Euchale]

Just adding myself in as well with this problem. Same boat, was able to use it fine, but can't use it anymore.

[PKK02]

In case people are still having this problem on Auto, I managed to use loras with Forge (https://github.com/lllyasviel/stable-diffusion-webui-forge). It's basically auto but much faster. It's apparently been abandoned but for what I do it works just fine. I can use multiple loras at the same time with no problems. There's an asterisk however. If I try to swap >checkpoints< in a session it still freezes my system so there's that.

[aurelion314]

In case people are still having this problem on Auto, I managed to use loras with Forge (https://github.com/lllyasviel/stable-diffusion-webui-forge).

Thanks for the suggestion! I've been watching this thread hoping for a fix, but until then it's nice to have another option.

[nekoworkshop]

Bumping this since I'm currently on dev branch (v1.9.4-169-ga30b19dd) and am seemingly experiencing similar issues.

I have 48GB of RAM so I didn't notice the growing leak until my C drive running out of space due to the page file growing to 60GBs. That's a nearly a 100GB commit limit.

Here are the numbers from VMMap.

When idling, the Working Set size is about 7GB. But the program holds a huge amount of memory committed.

[God-damnit-all]

This is getting buried when it's a severe issue. Desperation time. I apologize in advance:
@w-e-w @AUTOMATIC1111 @akx @catboxanon @KohakuBlueleaf @missionfloyd @light-and-ray

[wfjsw]

Please set environment variable PYTHONFAULTHANDLER=1 or use -X faulthandler and then trigger the error again. Upload the full log after crash, together with the dump file located in %LOCALAPPDATA%\CrashDumps.

[nailz420]

crash.txt
crash2.txt

Tell me what commands to execute on the crash dump. I don't want to share it.

Please set environment variable PYTHONFAULTHANDLER=1 or use -X faulthandler and then trigger the error again. Upload the full log after crash, together with the dump file located in %LOCALAPPDATA%\CrashDumps.

[wfjsw]

There is a high chance that I need more than it. Let's wait until someone shares.

[nailz420]

There is a high chance that I need more than it. Let's wait until someone shares.

Is there any way I can share it privately?

[wfjsw]

There is a discord channel on the wiki, if you want to join.

[nailz420]

There is a discord channel on the wiki, if you want to join.

I don't see any discord mention on A1111 wiki. However, I am on the stable diffusion discord. I don't see your (wfjsw) nickname there though.

[wfjsw]

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Contributing

It is hidden deep for some reason.

[aearone]

After the latest webui update I can confirm that the memory leak is fixed, the system does not consume more than 20-25gb of RAM when generating using LoRA and does not crash with an error.

[nailz420]

After the latest webui update

What latest update? What branch?

[aearone]

After the latest webui update

What latest update? What branch?

9e404c3

[nailz420]

modules/models/sd3/sd3_model.py Isn't this only for SD3?

…

On Wed, Jul 3, 2024 at 8:16 AM Nikolay L. ***@***.***> wrote: After the latest webui update What latest update? What branch? 9e404c3 <9e404c3> — Reply to this email directly, view it on GitHub <#15175 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGBCKPHCBJVHDS5QMD3IA6LZKPTSPAVCNFSM6AAAAABELRFZX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVHEZTSMRRGE> . You are receiving this because you commented.Message ID: ***@***.***>

[aearone]

modules/models/sd3/sd3_model.py Isn't this only for SD3?
…
On Wed, Jul 3, 2024 at 8:16 AM Nikolay L. @.> wrote: After the latest webui update What latest update? What branch? 9e404c3 <9e404c3> — Reply to this email directly, view it on GitHub <#15175 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGBCKPHCBJVHDS5QMD3IA6LZKPTSPAVCNFSM6AAAAABELRFZX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMBVHEZTSMRRGE . You are receiving this because you commented.Message ID: @.>

This update also extended to the main webui. At least my branch was updated when I started webui and the memory leak stopped when I used --medvram

[nailz420]

There haven't been any commits to the master branch since May 28, 2024. Last update on a branch was 4 days ago.

[aearone]

There haven't been any commits to the master branch since May 28, 2024. Last update on a branch was 4 days ago.

Yes, that's correct, and yet my webui got the updates when running with the git pull command.

[BlackWyvern]

git checkout'd back to master and did a git pull. I can confirm that SDXL models no longer use 24+GB of my ram whilst idle and try to crash my system on execute. And SD1.5 uses a 'mere' 10GB compared to the 18 it kept trying to allocate to.

But I also now am completely unable to switch models due to new cuda memory errors.. And SDXL fails to complete due to "Press any key to continue..." at 100% which event viewer points to an error in c10.dll. #15430

[God-damnit-all]

There haven't been any commits to the master branch since May 28, 2024. Last update on a branch was 4 days ago.

Yes, that's correct, and yet my webui got the updates when running with the git pull command.

Sounds like there might've been a dependency change. Could you list the output of dir /b (Windows) / ls -A1 --group-directories-first (Linux) in your venv's Libs/site-packages folder?

[nailz420]

I upgraded from 8 to 16 VRAM and removed the medvram-sdxl flag from the launcher command. I run git pull on every launch. I can still reproduce the memory issue with or without loras by running 1 batch gens (batch size 4-8) on them on different models.

[aearone]

There haven't been any commits to the master branch since May 28, 2024. Last update on a branch was 4 days ago.

Yes, that's correct, and yet my webui got the updates when running with the git pull command.

Sounds like there might've been a dependency change. Could you list the output of dir /b (Windows) / ls -A1 --group-directories-first (Linux) in your venv's Libs/site-packages folder?

F:\stable-diffusion-webui\venv\Lib\site-packages>dir /b
accelerate
accelerate-0.21.0.dist-info
aenum
aenum-3.1.15.dist-info
aiofiles
aiofiles-23.2.1.dist-info
aiohttp
aiohttp-3.9.5.dist-info
aiosignal
aiosignal-1.3.1.dist-info
altair
altair-5.3.0.dist-info
antlr4
antlr4_python3_runtime-4.9.3-py3.10.egg-info
anyio
anyio-3.7.1.dist-info
async_timeout
async_timeout-4.0.3.dist-info
attr
attrs
attrs-23.2.0.dist-info
blendmodes
blendmodes-2022.dist-info
certifi
certifi-2024.7.4.dist-info
CHANGELOG.md
charset_normalizer
charset_normalizer-3.3.2.dist-info
cleanfid
clean_fid-0.1.35.dist-info
click
click-8.1.7.dist-info
clip
clip-1.0-py3.10.egg-info
colorama
colorama-0.4.6.dist-info
contourpy
contourpy-1.2.1.dist-info
cv2
cycler
cycler-0.12.1.dist-info
dateutil
deprecation-2.1.0.dist-info
deprecation.py
diskcache
diskcache-5.6.3.dist-info
distutils-precedence.pth
einops
einops-0.4.1.dist-info
exceptiongroup
exceptiongroup-1.2.1.dist-info
facexlib
facexlib-0.3.0.dist-info
fastapi
fastapi-0.94.0.dist-info
ffmpy-0.3.2-py3.10.egg-info
ffmpy.py
filelock
filelock-3.15.4.dist-info
filterpy
filterpy-1.4.5-py3.10.egg-info
fontTools
fonttools-4.53.1.dist-info
frozenlist
frozenlist-1.4.1.dist-info
fsspec
fsspec-2024.6.1.dist-info
ftfy
ftfy-6.2.0.dist-info
functorch
git
gitdb
gitdb-4.0.11.dist-info
GitPython-3.1.32.dist-info
google
gradio
gradio-3.41.2.dist-info
gradio_client
gradio_client-0.5.0.dist-info
h11
h11-0.12.0.dist-info
httpcore
httpcore-0.15.0.dist-info
httpx
httpx-0.24.1.dist-info
huggingface_hub
huggingface_hub-0.23.4.dist-info
idna
idna-3.7.dist-info
imageio
imageio-2.34.2.dist-info
importlib_resources
importlib_resources-6.4.0.dist-info
inflection
inflection-0.5.1.dist-info
inflection.py
isympy.py
jinja2
jinja2-3.1.4.dist-info
jsonmerge
jsonmerge-1.8.0-py3.10.egg-info
jsonschema
jsonschema-4.23.0.dist-info
jsonschema_specifications
jsonschema_specifications-2023.12.1.dist-info
kiwisolver
kiwisolver-1.4.5.dist-info
kornia
kornia-0.6.7.dist-info
lark
lark-1.1.2.dist-info
lazy_loader
lazy_loader-0.4.dist-info
lightning_fabric
lightning_utilities
lightning_utilities-0.11.3.post0.dist-info
llvmlite
llvmlite-0.43.0.dist-info
markupsafe
MarkupSafe-2.1.5.dist-info
matplotlib
matplotlib-3.9.1.dist-info
matplotlib.libs
mpl_toolkits
mpmath
mpmath-1.3.0.dist-info
multidict
multidict-6.0.5.dist-info
multipart
networkx
networkx-3.3.dist-info
numba
numba-0.60.0.dist-info
numpy
numpy-1.26.2-cp310-cp310-win_amd64.whl
numpy-1.26.2.dist-info
numpy.libs
nvfuser
omegaconf
omegaconf-2.2.3.dist-info
opencv_python-4.10.0.84.dist-info
open_clip
open_clip_torch-2.20.0.dist-info
orjson
orjson-3.10.6.dist-info
packaging
packaging-24.1.dist-info
pandas
pandas-2.2.2.dist-info
pandas.libs
piexif
piexif-1.1.3.dist-info
PIL
Pillow-9.5.0.dist-info
pillow_avif
pillow_avif_plugin-1.4.3.dist-info
pip
pip-22.2.1.dist-info
pkg_resources
protobuf-3.20.0-py3.10-nspkg.pth
protobuf-3.20.0.dist-info
psutil
psutil-5.9.5.dist-info
pydantic
pydantic-1.10.17.dist-info
pydevd_plugins
pydub
pydub-0.25.1.dist-info
pylab.py
pyparsing
pyparsing-3.1.2.dist-info
python_dateutil-2.9.0.post0.dist-info
python_multipart-0.0.9.dist-info
pytorch_lightning
pytorch_lightning-1.9.4.dist-info
pytz
pytz-2024.1.dist-info
pywavelets-1.6.0.dist-info
pywt
PyYAML-6.0.1.dist-info
README.md
referencing
referencing-0.35.1.dist-info
regex
regex-2024.5.15.dist-info
requests
requests-2.32.3.dist-info
resize_right
resize_right-0.0.2.dist-info
rpds
rpds_py-0.19.0.dist-info
safetensors
safetensors-0.4.2.dist-info
scikit_image-0.21.0.dist-info
scipy
scipy-1.14.0-cp310-cp310-win_amd64.whl
scipy-1.14.0.dist-info
scipy.libs
semantic_version
semantic_version-2.10.0.dist-info
sentencepiece
sentencepiece-0.2.0.dist-info
setuptools
setuptools-69.5.1.dist-info
six-1.16.0.dist-info
six.py
skimage
smmap
smmap-5.0.1.dist-info
sniffio
sniffio-1.3.1.dist-info
spandrel
spandrel-0.1.6.dist-info
starlette
starlette-0.26.1.dist-info
sympy
sympy-1.13.0.dist-info
test
tifffile
tifffile-2024.7.2.dist-info
timm
timm-1.0.7.dist-info
tlz
tokenizers
tokenizers-0.13.3.dist-info
tomesd
tomesd-0.1.3.dist-info
toolz
toolz-0.12.1.dist-info
torch
torch-2.1.2+cu121.dist-info
torchdiffeq
torchdiffeq-0.2.3.dist-info
torchgen
torchmetrics
torchmetrics-1.4.0.post0.dist-info
torchsde
torchsde-0.2.6.dist-info
torchvision
torchvision-0.16.2+cu121.dist-info
tqdm
tqdm-4.66.4.dist-info
training
trampoline
trampoline-0.1.2.dist-info
transformers
transformers-4.30.2.dist-info
typing_extensions-4.12.2.dist-info
typing_extensions.py
tzdata
tzdata-2024.1.dist-info
urllib3
urllib3-2.2.2.dist-info
uvicorn
uvicorn-0.30.1.dist-info
wcwidth
wcwidth-0.2.13.dist-info
websockets
websockets-11.0.3.dist-info
xformers
xformers-0.0.23.post1.dist-info
yaml
yarl
yarl-1.9.4.dist-info
_distutils_hack
_yaml
__pycache__

F:\stable-diffusion-webui\venv\Lib\site-packages>