Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queue Errors -> Session Hijacking #713

Closed
666spawn999 opened this issue Oct 17, 2023 · 6 comments · Fixed by #981
Closed

Queue Errors -> Session Hijacking #713

666spawn999 opened this issue Oct 17, 2023 · 6 comments · Fixed by #981
Labels
bug Something isn't working
Milestone

Comments

@666spawn999
Copy link

666spawn999 commented Oct 17, 2023

Hi,
I'm trying to setup a Fooocus intance for me and my colleagues.
Sadly, a problem with the queuing System is preventing, that we can use it together.

Error:

  • Client A is generating an image and exits it's browser session during generation (sure, this isn't wanted, but it happens)
  • Client B, who is waiting in the queue, gets the preview and results from the running generation of Client A
  • while getting the results the progress bar is really fast, because the frames are already generated and just buffered...
  • all following Tasks are lagging one result because the queue mixed up the resolve of the queue/clients
  • this persists indefinitely and the service needs to be restarted to fix the queue.

Video of Problem with 2 browsers
Fooocus_Queue_Hijacking2

Steps to Replicate:

  1. Open 2 Browser windows (same/different Browsers or incognito, doesn't matter)
  2. start a job in each window -> Client A generates, Client B is in queue
  3. Exit or reload Client A Windows -> Client B fills the slot and gets the results of ClientA instead of it's own task!
  4. The queue is from now on mixed up. Each Generation shows the Results of the Task before.
  5. Server Restart is needed to fix

some more thoughts

  • I have already looked into this a little bit, and I think the Gradio queue and WebSocket is involved in this.
  • The Websocket Session_Ids are different in the browsers. But when ClientA quits, the rest of the result "stream" is send to the other connected websocket, even if the ID is different.
  • Adding a check, that only the Websocket who started the task, should get results, could be a fix?

Altogether this is a big problem that currently makes it impossible to share Fooocus Installation with others. All Forks I have tried are having the same problem. Some even more because they send Results to different clients in sync..

I think, this issues are related:
#625

A fix would be highly appreciated!
If you have some tips on this, I can also try to fix it myself, but currently I'm a little lost at this issue.

If this is a general problem with gradio and can't be fixed by the Fooocus Team maybe someone has some idead how to fix it in gradio?

(I really love Fooocus and all the features and the simplicity of it. Thanks for all your work!
But sadly, bugs like this prevent me from using it in production or teams :( )

@adelorenzo-portainer
Copy link

yup, I am experiencing the same problem.

@666spawn999
Copy link
Author

This other issues seem to be the same bug
#501
#481

@yllwboy
Copy link

yllwboy commented Oct 22, 2023

I've experienced the same issue, as well as encountering this issue with a single client if the client exits its browser session.

@lllyasviel
Copy link
Owner

a temp fix is landed in 2.1.751

@666spawn999
Copy link
Author

That fix makes it absolutely more tolerable because the queue doesn't break anymore.
Thanks for looking into it!

The only problem that is left is:

  • when client A quits its session while processing an image,
  • client B, that is in the queue gets his results AND after that his own results from the queue.

As said, the queue doesn't break anymore. But because of privacy reasons a fix for this possible image-leak-bug is still relevant I think.

But so far, great work! And thanks very much, because I don't have to restart the server anymore!
:D

@mashb1t
Copy link
Collaborator

mashb1t commented Oct 31, 2023

@666spawn999 tried my best as a python newbie, fixed in #826

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
5 participants