ctransformers: move thread and seed parameters

[cal066]

Make it more in line with webui's way of managing loaders.

While ctransformers can have the number of threads and seed set during the loading and generation phase, set the threads during model load and seed during generate phase to align with llama.cpp expectations.

Checklist:

• I have read the Contributing guidelines.

[oobabooga]

Thank you. I confirm that the threads option works this way, but the seed seems to have no effect -- if I generate twice for the same prompt but different seeds, I get two different responses. Is that a bug in ctransformers?

[bartowski1182]

if I generate twice for the same prompt but different seeds, I get two different responses.

A typo? I presumed that was intended behaviour

[oobabooga]

Yes, I meant same seeds*

[cal066]

@oobabooga in ctransformers ctransformers/llm.py:

    def sample(...)
       ...
        seed = get(seed, config.seed)

        return self.ctransformers_llm_sample(
            top_k,
            top_p, 
            temperature,
            repetition_penalty,
            last_n_tokens,
            seed,
        )

Eventually it makes its way to the LLM::Sample C++ class models/llm.h:

    if (seed < 0) {
      seed = time(nullptr);
    }
    std::mt19937 rng(seed);

From what I understand, as long as seed is -1, it should be random.
@marella Can you comment?
@oobabooga What model were you testing?

[oobabooga]

I tried falcon-7b-instruct.ggccv1.q4_0.bin from here: https://huggingface.co/TheBloke/falcon-7b-instruct-GGML/tree/main

Make sure to set a high temperature/top_k/top_p to make the replies unpredictable.

[cal066]

From my testing, it seems to be working (using the same model and simple-1 Parameters):

Instruction: Roll a 20-sided dice.
Output: The result of rolling a 20-sided dice is 1.
Output: The result of rolling a 20-sided dice is 20.
Output: The result of rolling a 20-sided dice is 11.

[marella]

From what I understand, as long as seed is -1, it should be random.

Yes, seed=-1 means it will use a random seed.

Is shared.settings['seed'] a constant or can be changed by user from UI? Shouldn't you read it from a user configurable param like shared.args.llama_cpp_seed?

[oobabooga]

Well noted @marella. The UI parameters are passed in the state dict. I have made this fix, so it should be good now.

Thanks @cal066 for the PR and thanks @marella for the feedback (and for ctransformers :).

[marella]

Please use self.model() as self.model._stream() is an internal method and should not be used directly.