KoboldCPP-v1.82.1.yr0-ROCm

[YellowRoseCx]

---

This discussion was created from the release KoboldCPP-v1.82.1.yr0-ROCm.

[TheLapinMalin]

Hi,

I'm seeing this error with a 6750XT. I read here (#80) that this popped up a few months ago, but is it still an issue for anyone else?

ROCm error: CUBLAS_STATUS_INTERNAL_ERROR
  current device: 0, in function ggml_cuda_mul_mat_batched_cublas at D:/a/koboldcpp-rocm/koboldcpp-rocm/ggml/src/ggml-cuda/ggml-cuda.cu:1719
  hipblasGemmBatchedEx(ctx.cublas_handle(), HIPBLAS_OP_T, HIPBLAS_OP_N, ne01, ne11, ne10, alpha, (const void **) (ptrs_src.get() + 0*ne23), HIPBLAS_R_16F, nb01/nb00, (const void **) (ptrs_src.get() + 1*ne23), HIPBLAS_R_16F, nb11/nb10, beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne01, ne23, cu_compute_type, HIPBLAS_GEMM_DEFAULT)
D:/a/koboldcpp-rocm/koboldcpp-rocm/ggml/src/ggml-cuda/ggml-cuda.cu:73: ROCm error

PS: For info, v1.80.3 also fails for me. The latest release that works is v1.79.1.yr1.

[RageshAntonyHM]

@YellowRoseCx

Is possible to run DeepSeek R1 in this repo https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q8_0 using koboldcpp-rocm on the runpod's docker image runpod/pytorch:2.0.1-py3.10-rocm5.7-ubuntu22.04 ?

I am thinking to use 8x MI300X

[Danik-droid]

Not using MMAP doesn't make sense, often GPU caches something to RAM while having free VRAM and everything runs slower. Example with MMAP enabled I can easily use Mistral Small Q4L with 16k context and 8bit kvcache everything in GPU and it runs lightning fast and takes up 15.7GB of VRAM (in total). Without MMAP the model takes up over 17GB of VRAM (in total) including about 15GB in card memory and about 2GB buffered in RAM. It runs slowly, processing is terribly slow... it makes no sense.

[DistyPoo]

I assume this doesn't support deepseek r1 yet

[TheLapinMalin]

@YellowRoseCx Thanks for releasing v1.82.4 with ROCm 5.7. But unfortunately, it doesn't seem to run on my end and fails with this error:

***
Welcome to KoboldCpp - Version 1.82.4.yr0-ROCm
For command line arguments, please refer to --help
***
Unable to detect VRAM, please set layers manually.
Auto Selected HIP Backend...

Auto Recommended GPU Layers: 35
Initializing dynamic library: koboldcpp_hipblas.dll
Traceback (most recent call last):
  File "PyInstaller\loader\pyimod03_ctypes.py", line 53, in __init__
  File "ctypes\__init__.py", line 374, in __init__
OSError: [WinError 127] The specified procedure could not be found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "koboldcpp.py", line 6117, in <module>
  File "koboldcpp.py", line 5595, in main
  File "koboldcpp.py", line 463, in init_library
  File "PyInstaller\loader\pyimod03_ctypes.py", line 55, in __init__
pyimod03_ctypes.install.<locals>.PyInstallerImportError: Failed to load dynlib/dll 'C:\\Users\\xxx\\AppData\\Local\\Temp\\_MEI80642\\koboldcpp_hipblas.dll'. Most likely this dynlib/dll was not found when the application was frozen.
[PYI-20880:ERROR] Failed to execute script 'koboldcpp' due to unhandled exception!```

[YellowRoseCx]

What version of Python do you have on your computer?

[TheLapinMalin]

What version of Python do you have on your computer?

Running 3.10.6 (and never had this issue running KoboldCPP until just now).

[GramSeco]

Same error, also have a 6750 XT. The latest version that works is the 79.1.yr1 fix. Thank you to YellowRoseCx for all their time and effort!

[Ugijago]

hello first timer here , dont work dunno why crash immediately when launch with rocm but vulkan work, try older mention in some thread 1.79 xxx yr1 and its work fine, is there any fix? i use 6600 rocm is faster than vulkan , i am in window

[TheLapinMalin]

@YellowRoseCx I tried your latest 1.83.1 release, but unfortunately get the same issue. :(

ROCm error: CUBLAS_STATUS_INTERNAL_ERROR
  current device: 0, in function ggml_cuda_mul_mat_batched_cublas at D:/a/koboldcpp-rocm/koboldcpp-rocm/ggml/src/ggml-cuda/ggml-cuda.cu:1805
  hipblasGemmStridedBatchedEx(ctx.cublas_handle(), HIPBLAS_OP_T, HIPBLAS_OP_N, ne01, ne11, ne10, alpha, (const char *) src0_f16, HIPBLAS_R_16F, nb01/nb00, nb02/nb00, (const char *) src1_f16, HIPBLAS_R_16F, nb11/nb10, nb12/nb10, beta, ( char *) dst_t, cu_data_type, ne01, nb2/nb0, ne12*ne13, cu_compute_type, HIPBLAS_GEMM_DEFAULT)
D:/a/koboldcpp-rocm/koboldcpp-rocm/ggml/src/ggml-cuda/ggml-cuda.cu:75: ROCm error

Is there anything I can do to help troubleshoot this?