System Backend

[outlookhazy]

Disclaimer: I'm not a Python developer, so I may be missing some subtleties

Currently, it appears that things are very coupled to the UI. Both gradio and streamlit seem to be rpc wrappers that are calling the inference functions directly. From a design perspective, it would make sense to me for there to be a generic backend 'engine/server' abstracted away from the UI that handles loading models, tracking sessions, queuing and dispatching inference (including abstracting away different underlying libraries), etc.
Rather than the frontends calling into the library directly, they would make requests into the local server/engine which would allow them to share a lot of the logic.
The primary benefit I see (after code consolidation) would be to allow concurrent sessions, especially from multiple UIs. The jobs-manager is already attempting to do part of what I'm suggesting, but as-is it's broken for simultaneous sessions. A major bonus in my mind is that this would also probably make it trivial to add in a generic API endpoint that would allow both existing web clients, as well as custom clients in other languages to all coexist and leverage all the good work being put into the features behind the scenes.

[outlookhazy]

The Gradio frontend without jobs-manager deals with simultaneous clients OK; one will just sit at loading while gradio moves through its own queue. Streamlit (which the dev branch seems to indicate is the future) looks great so far with the preview updates, but seems to have big issues with multiple sessions. It looks like the entire model attempts to be reloaded for each new client, which runs out of memory and crashes. One client can then reload the browser and get it to go again, but not a great experience.