[PR?] Fill in the middle completer

[NightMachinery]

I have written a usable fill-in-the-middle function. The code is available here (look for night/ellama-code-fill-in-the-middle), and I can clean it up further for inclusion in ellama, if there's interest. Or someone else can do that.

Here are some demos with llama3-70b-8192:

ellama_fill_middle_v2.mp4

ellama_fill_middle.mp4

The highlights will be cleared the next time you press C-g, though this depends on Doom's doom-escape-hook; I couldn't find a built-in hook for keyboard-quit.

[s-kostyaev]

Hi @NightMachinery
Looks interesting.
Have you signed FSF papers? If so, open PR 🙂

[NightMachinery]

I have signed the copyright papers for emacs before; are these the same?

I'll use this function for a while to see how robust it is. Its implementation is very hacky currently, and the heuristics used look somewhat model-dependent.

[s-kostyaev]

I have signed the copyright papers for emacs before; are these the same?

Yes. It means you don't need to do it second time.

I'll use this function for a while to see how robust it is. Its implementation is very hacky currently, and the heuristics used look somewhat model-dependent.

we can improve it during code review. And it will be model dependent, because different models handle FIM differently, has different tags etc.

[NightMachinery]

different models handle FIM differently

The current implementation is a hack that uses normal LLMs tuned for chat. If the model supported FIM, the implementation would become a lot simpler. I am not aware of any FIM API though. (Surely there must be some open models, but is there anything competent?)

[s-kostyaev]

See codeqwen, deepseek coder (not latest one) and Starcoder 2 7b (in 15b fim is broken).

[NightMachinery]

See codeqwen, deepseek coder (not latest one) and Starcoder 2 7b (in 15b fim is broken).

Is there any cloud API for these models? (e.g., on OpenRouter or OpenAI)

I think the llm backend doesn't support FIM completion, either.

[s-kostyaev]

I don't think llm should do something to add fim support, it's about right prompt format.
I don't know about cloud APIs, I don't use it.