Please update llama.cpp

[rekire]

Please update the project to make it compatible again with the latest releases.
I already created a script which updated the ffi code, but my knowledge about llama.cpp is too bad to fix the issues myself.

In the mean time I managed it to automate those changes. So you can inspect the roll outs here.

[rekire]

I managed it, that my fork compiles with one of the latest builds (b4586). However I still don't understand some parts so I commented it out without really knowing what I broke, like:

    lib.llama_sampler_chain_add(
        _smpl,
        lib.llama_sampler_init_penalties(
            lib.llama_n_vocab(model!),
            lib.llama_token_eos(model!),
            lib.llama_token_nl(model!),
            samplerParams.penaltyLastTokens,
            samplerParams.penaltyRepeat,
            samplerParams.penaltyFreq,
            samplerParams.penaltyPresent,
            samplerParams.penaltyNewline,
            samplerParams.ignoreEOS));

and

      lib.llama_sampler_chain_add(
          _smpl,
          lib.llama_sampler_init_dry(
              vocab!,
              samplerParams.dryPenalty,
              samplerParams.dryMultiplier,
              samplerParams.dryAllowedLen,
              samplerParams.dryLookback,
              seqBreakersPointer,
              numBreakers));

I have no idea what that does and how to fix it.

The cropped output from #57 is still present and it does not crash. I would be happy to get some feedback.

In order to test my changes you can change your pubspec.yaml to this:

dependencies:
  llama_cpp_dart:
    git:
      url: https://github.com/rekire/llama_cpp_dart_fork.git
      ref: 0.0.9-dev-b4586

Please note it is not required to build llama.cpp you can simply download the .so/.dylib/.dll from https://github.com/ggerganov/llama.cpp/releases/tag/b4586 (not tested with Android or iOS)

[RojasFD007]

Hi, I tried your build on my Pixel 6A and it works, I just have to place the .so file in the jniLibs folder and add the following lines in the build.gradle externalNativeBuild { cmake { cppFlags "" arguments "-DANDROID_STL=c++_shared" }

[netdur]

I am working on scripts to update with llama.cpp --- if all goes well, this weekend

[rekire]

Feel free to check my build scripts they might be helpful: https://github.com/rekire/llama_cpp_dart_fork/tree/main/.github

[netdur]

@rekire thank you sir, that's awesome