Related to stop prompt

[hlhr202]

Hi, some models got very stupid stopping logic (eg. vicuna under instruct mode)
llama-rs can provide a way (like a stop prompt argument) to stop it by giving a text sequence.
What I v investigated are as follows:

https://github.com/Atome-FE/llama-node/blob/main/packages/llama-cpp/src/llama.rs#L152
https://github.com/sobelio/llm-chain/blob/main/llm-chain-llama/src/executor.rs#L96

I think llama.cpp also provide something called reverse prompt or anti prompt (but they just use it in interactive mode)

[LLukas22]

I agree that adding stop word/sequence support natively would be helpful for using models in a chatbot setting, especially ones that aren't specifically trained for that purpose. Perhaps we could add a list of strings to the InferenceParameters, tokenize them, and then match the last N-generated tokens against the tokenized stop words. This should be easy enough and would prevent models from talking to themselves 😓.

[danforbes]

Here is a naive implementation https://github.com/danforbes/llama-rs/tree/dfo/feat/chat

[philpax]

@danforbes Sorry, keep forgetting to get back to you on this! Yeah that seems fine to me, want to add antiprompt: Option<&str> to inference_with_prompt?

[danforbes]

You need to provide more information than just the "reverse prompt" - among other things you need to provide some kind of callback for receiving user input. There are also some changes to the way that returned tokens are handled (EOT handling as well as reverse prompt handling) that I wasn't sure how to implement cleanly in the existing inference_with_prompt function.

Edit: Also, the existing inference_with_prompt function already has too many arguments so I'm very reluctant to add even more. In this branch I actually took a stab at condensing the args to the inference function. https://github.com/danforbes/llama-rs/tree/dfo/feat/chat

[danforbes]

@hlhr202 this should be implemented with #206, but please open another Issue if you have more suggestions 🙏🏻