bug: Jan doesn't auto detect GPU on Windows

[hahuyhoang411]

Jan Windows v0.4.0-41
CPU: i7-12700
CUDA: 12.3 Driver Version: 545.92
RAM 32GB

Describe the bug
Jan will run on the CPU on my computer by default. I and @vuonghoainam did a quick sync on this. It turns out I can't run the Nitro GPU because of it's missing cudart

Screenshots

[hiro-v]

The solution is:

• Nitro extension onLoad to download respective cuda runtime from https://github.com/janhq/nitro/releases/tag/v0.1.27 (pls change with version in bin/version.txt). Please only install CUDA 12 for now, no need to install cuda 11, we can fix later.
  
• Unzip in the bin/win-cuda/ with flat file
• Done, this will make sure that:
  • Nitro running is decoupled with OS cuda toolkit
  • Any incompatibility
  • The problem that may arise with cuda toolkit on windows/ linux that even CUDA is there but cannot access
• Use Rex laptop to test.

[louis-menlo]

Rescheduled to 0.4.3

[freelerobot]

Is this fixed? @hiento09 If so, can you close this issue?

[hiento09]

Not yet @0xSage, I may need to re-schedule this bug to next sprint since it relates to nvidia driver + cuda dependency/compability.

[hiento09]

Subtask:

• feat: Research a new approach for nitro cuda dependencies for windows and linux cortex.cpp#280
• Integrate to Jan for downloading dependencies process

[hiento09]

For the GPU detection issue, the current approach for the Jan app is as suggestion bellow:

1. In Nitro inference plugin, try execute the nvidia-smi command to check for the presence of a GPU. If a GPU is detected, proceed to verify CUDA. This involves checking if CUDA's DLL files exist in the environment. If they do, Nitro will automatically start.

If no GPU is found, a warning will be automatically issued to the user, indicating the absence of a GPU.

2. Warning Message Contents:

• The warning message will include the following:
  • Instructions on installing NVIDIA drivers directly from Jan's official guidelines, tailored to the CUDA version used for building Nitro, and guidance on building from source for users.
  • Steps to replace the Nitro binary file within the Jan app, with a caution that building from source will not include Jan's Windows code signing.

3. Links and Resources:

• The warning will also provide a link to the official CUDA installation page.
• A CUDA and NVIDIA driver compatibility matrix will be included for user reference.

Additional Notes:

• An option to manually check NVIDIA and GPU status might be added to the guidance bar, if needed.
• If a GPU is detected, an option to start Nitro specifying which GPU(s) to use (either one or up to three) can be considered for future implementation.

[tikikun]

I think we should switch to an explicit GPU support by having something like a "Hardware acceleration" dropdown (Jellyfin example in the image) to reduce edge case, rather than automatic.
-Jan will try to find the most suitable option for the dropdown

• If not succeed in automation, then user specify their hardware acceleration.

There are many other edge cases to solve for and it's very high chance that the hardware owner knows what hardware they're having.

[tikikun]

I created an issue here for this feature request for tracking
#1208