Notes

[Amranax]

This thread is purely for tracking general information and research.

[Amranax]

Youtube Videos

Deepseek R1 Innovation Summary
Deepseek V3 in-depth review
Visualized math behind Query,Key,Value matrices
Intro to MoEs

[Amranax]

Papers on RoPE

1. RoFormer

Papers on MLA

1. TransMLA: Multi-head Latent Attention Is All You Need
2. DeepSeek-V2

Papers on MoE

1. Mixtral of Experts
2. Switch Transformers
3. DeepSeekMoE
4. Original Paper

Papers on MoE parameter prediction (from proff Piazza)

1. fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving
2. DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
3. HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference
4. MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services
5. Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

[mauer4]

Potentially for both of us to read:
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Medium - DeepSeek-V3 Explained 1: Multi-head Latent Attention
Medium - Multi-head Latent Attention (MLA): Secret behind the success of DeepSeek Large Language Models

RoPE: specific:
RoFormer: Enhanced Transformer with Rotary Position Embedding

Low Level Optimizations:
Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning
FlashAttention 3
Efficient Large Language Models: A Survey Especially the "Efficient Inference Section"
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
Flux: Fast Software-based Communication Overlap on GPUs through Kernel Fusion

[mauer4]

Relevant github Issue:
perf: improve plan performance by using non-blocking memcpy #547
feat: support MLA decode #551
Support MLA (Multi-Head Latent Attention) in DeepSeek-v2
Keywords: Pre-fill kernel, refactor comon prefill kernel, CUTE

[mauer4]

I feel like the goal of milestone 1 is distilling in my head to the following 2 tasks:

1. Determine the MLA (+RoPE) / MoE computation graph in depth, including linear algebra mathematical representation
2. Survey Inference optimizations and low level techniques

These two tasks should produce for us by the end of milestone 1 a narrowed-down algorithmic section that we are 'attacking' and the specific techniques we propose to use for the optimization.