Performance decreased severely

[TheOnlyWiseJEDI]

Using: text-generastion-webui
Model: ehartford_dolphin-2.2-mistral-7b - not quantized
Installed: using pip install -r requirements
OS: Windows 10
Shell: cmd.exe
Python: Python 3.11.3
pymemgpt installed: yes
Hardware: 12th Gen i7 64GB/RAM, RTX3060 12GB RAM
LLM and MemGPT on same system

I have recently reinstalled memgpt so all is fresh and up to date. I have NOT altered any settings in text-generation-webui. This problem existed before and after a reinstall.

There is a significant performance decrease since a recent update. 'reporter' said he thought it might be the grammar update and that he noticed it also. What used to be a processing time of 4 - 12 seconds has become more often than not, 20+ seconds. Even as much as 39 seconds.

With the model loaded and memgpt running these results are consistent through the CLI however if I flip to the webui interface and pose the same question the performance time is more than 1/2 faster than when done through memgpt. This was not true last week (presumably prior to whatever update changed things).

[cpacker]

@TheOnlyWiseJEDI can you try disabling grammar by forcing a specific wrapper?

If you pass --model airoboros-l2-70b-2.1, it shouldn't use grammar, so things should be as they were before (unless there's something I'm missing).

(If you don't pass any model, the default will be grammar-based)

[TheOnlyWiseJEDI]

It is no longer possible to use --model when using memgpt with 'run'  

[TheOnlyWiseJEDI]

I tried to include model when creating a new agent - same result

…

On Mon, Nov 6, 2023 at 4:05 PM Charles Packer ***@***.***> wrote: @TheOnlyWiseJEDI <https://github.com/TheOnlyWiseJEDI> can you try disabling grammar by forcing a specific wrapper? If you pass --model airoboros-l2-70b-2.1, it shouldn't use grammar, so things should be as they were before (unless there's something I'm missing). — Reply to this email directly, view it on GitHub <#331 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5QYLAREUMC7ENB34FY2BMLYDFGJJAVCNFSM6AAAAAA7AAMNI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJWGQ2DEMJYGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[TheOnlyWiseJEDI]

it looked like the images didn't get included when I replied from email, not sure if you could see them, I edited and reapplied the images, they should be available now

[cpacker]

It is no longer possible to use --model when using memgpt with 'run'

@vivi @sarahwooders can we do a hotpatch on this?

[vivi]

@TheOnlyWiseJEDI For now you can run memgpt run --model airoboros-l2-70b-2.1 and hit 'N' when asked "Would you like to select an existing agent?"

Hmm -- just saw your comment that you tried to do that. Looking into this now.

[vivi]

@TheOnlyWiseJEDI I can't seem to reproduce your bug on 0.1.18a1. Could you run pip show pymemgpt and if the version is <0.1.18a1, run pip install pymemgpt -U --pre?

I do think that you should be able to reload the same agent with the same wrapper so I have opened #338.

[TheOnlyWiseJEDI]

ok done. I had 0.1.17, now updated. I did this early this morning, was there another release today? What should be fixed now in this update?

…

On Mon, Nov 6, 2023 at 5:26 PM Vivian Fang ***@***.***> wrote: @TheOnlyWiseJEDI <https://github.com/TheOnlyWiseJEDI> I can't seem to reproduce your bug on 0.1.18a1. Could you run pip show pymemgpt and if the version is <0.1.18a1, run pip install pymemgpt -U --pre? — Reply to this email directly, view it on GitHub <#331 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5QYLAREBJW7GIY5FIEJ3JDYDFP2RAVCNFSM6AAAAAA7AAMNI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJWHA4TCNJWGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sarahwooders]

If you run pip install pymemgpt -U --pre, you should be able to have the --model flag without an error, as long as it matches the model of the agent you are resuming. You don't need to specify any additional flags if you're resuming an existing agent.

[TheOnlyWiseJEDI]

I can't run memgpt at all now, even with just "memgpt run" - take a look at the other ticket you created

…

On Mon, Nov 6, 2023 at 8:30 PM Sarah Wooders ***@***.***> wrote: If you run pip install pymemgpt -U --pre, you should be able to have the --model flag without an error, as long as it matches the model of the agent you are resuming. You don't need to specify any additional flags if you're resuming an existing agent. — Reply to this email directly, view it on GitHub <#331 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5QYLATL3HYQY4DUDFZV3BTYDGFJPAVCNFSM6AAAAAA7AAMNI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXGEZDMNZUGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cpacker]

Closing this particular issue because it should be resolved in the update that just went live (0.1.18), but @TheOnlyWiseJEDI feel free to reopen if it's still happening or open a new issue if something else is wrong.