epic: Jan supports multiple CPU/GPU Hardware optimizations

[tikikun]

Problem

Windows users encounter elevated CPU usage (ex: #887), prompting the need for hardware optimizations in Nitro/Jan across various Windows architectures (CPU, GPU).

Success Criteria

• Achieve compatibility for Jan on Windows with 16GB of RAM, accommodating systems lacking a GPU.
• Enable Jan to operate seamlessly on Windows systems featuring Nvidia GPUs.

Out-of-scope

Not support Intel, AMD GPU for now, although we still detect these device.

Tasklist

Design & FE

• feat: add GPU acceleration in advanced settings #1646

Implementation

• feat: Add NVIDIA GPU monitoring on windows/ linux #286
• feat: Support WSL on windows #912
• epic: support AMD using Vulkan #913
• feat: Inference engine for Intel iGPU and intel GPU #915
• feat: optimization for oneAPI on windows #911
• feat: AMD CPU for Windows #914
• feat: User Selectable GPUs and GPU-based Model Recommendations #1730

Additional context

• Edge inferencing on windows is very important, Intel and Nvidia hardware having multiple optimizations that can be tuned to make LLM inference possible on windows.
• https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html

[tikikun]

https://www.npmjs.com/package/systeminformation

[tikikun]

Intel optimization:
Instructions set:
AVX2 - Consumer grade
AVX512 - Before gen 12 or Consumer grade
AVX_VNNI - Consumer grade with OneAPI
AVX512_VNNI - Server grade

[tikikun]

Windows WSL
#912

oneAPI for Windows
#911

AMD GPU for Windows
#913

AMD CPU for Windows
#914

Intel GPU (maybe crossing with oneAPI)
#915

[dan-menlo]

@tikikun I've renamed this Epic and added more detail to the "Top" post, please take a look.

I will be creating a separate "Jan should work on Windows" epic, that will be more Product-focused (there will be some overlap)

[imtuyethan]

Archive the original comment of Alan:

---

Problem

• Windows users experience 100% CPU usage (bug: Running neural-chat-7b consuming 100% CPU #887)
• Nitro/Jan needs to support hardware optimizations for different Windows architectures (CPU, GPU)

Success Criteria

• Jan can run on Windows with 16gb of RAM, no GPU (both Intel and AMD processors)
• Jan can run on Windows with Intel, AMD, Nvidia GPUs

Tasks

• feat: Support WSL on windows #912
• feat: optimization for oneAPI on windows #911
• epic: support AMD using Vulkan #913
• feat: AMD CPU for Windows #914
• feat: Inference engine for Intel iGPU and intel GPU #915

Additional context

• Edge inferencing on windows is very important, Intel and Nvidia hardware having multiple optimizations that can be tuned to make LLM inference possible on windows.
• https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html

[dan-menlo]

@Van-QA to decompose this into the engineering: Jan Inference Layer roadmap

[Van-QA]

Only 1 feat and 1 epic left, I will close this epic once the remaining ticket #915 completed.
On the other hand, I will follow up with the epic #913 in the engineering: Jan Inference Layer roadmap

[Van-QA]

as discussed with @hiro-v, the remaining feat #915 is nice to have, and will handled in the follow up epic #2092