-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
nKeep Persistent Context + Persistent Context Token Counts #62
nKeep Persistent Context + Persistent Context Token Counts #62
Conversation
… keep-n-persistent-context-+-persistent-context-token-counts
… keep-n-persistent-context-+-persistent-context-token-counts
All right, I fixed my botched merge. Should be ready for review again. |
A bit late to realize, but am I misunderstanding the functionality of n-keep? Can the memory even be "flushed" out of the context like I thought it could? If not, a lot of this PR could be scrapped. |
Afaik, n_keep is just what we are already doing but more accurate (i.e., it keeps n tokens at the beginning of the context). No backend will 'flush the context' as long as the prompt is within the context window. |
Nah, I think you must be imagining things. As far as I remember, the only changes I made to the code related to WI were when I moved some code to the WorldInfoModal and the draft PR #63. But neither change really altered how WI works in any way. |
Well, that's an afternoon of coding down the drain. The Token Counts for Memory, AN and WI could still be useful, but frankly, they don't seem all that important to me.
That's probably what I was thinking of then. |
Built on #56
Made memory and wi use n-keep for llama.cpp and kobold.cpp so it won't get flushed out of context. Since the opportunity arose, I also added token counts for the persistent context fields.