[FR] Mistral-7B Sliding Window Attention support

[wizzard0]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

The Mistral model https://mistral.ai/news/announcing-mistral-7b/ is fully supported with llama.cpp.

🟢 Uses Grouped-query attention (GQA) for faster inference

❌ Uses Sliding Window Attention (SWA) to handle longer sequences at smaller cost

"Outperforms Llama 2 13B on all benchmarks"

Current Behavior

Only 4K of context is available to use

Environment and Context

llama.cpp ac43576 (Sep 27)

[BarfingLemurs]

https://github.com/mistralai/mistral-src

[choltha]

I am surprised it works at all given it was trained on a partly different attention architecture.

Did you run these models?
https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF
https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF

[ggerganov]

Superseded by #3377

[wizzard0]

@choltha yes, and they work