Predicting activation patterns using a smaller model in order to optimise memory usage.

[smntlbt]

I wondered if it would be possible to:

1. Do an analysis of activation patterns as a function of prompts so that we could segment memory into logical units so that there's high probability of this memory being used together.

2. Given the output of this analysis. Train a smaller model, to predict given a prompt. which chunks of memory are to be used.

We could even run this model in parallel as to know through the course of execution, if we need to pre-fetch some chunks, before they are used.

Now we are obviously gonna have a bad prediction from time to time, so this would need to be handled.