10 free resources to learn LLM inference engineering

Covering topics such as:

  • KV cache and batching
  • Serving engines
  • GPU kernels and benchmarks

Start here and go in order:

1) LLM Inference Handbook (Modular)
↳ Learn the words first: prefill, decode, KV cache, latency, throughput.
↳ A reference you keep open while you read the rest.
https://handbook.modular.com/

2) Continuous batching from first principles (Hugging Face)
↳ Starts at attention and KV caching, ends at continuous batching.
↳ Shows why mixing prefill and decode in one batch helps throughput.

Continuous batching from first principles

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

3) Stanford CS336, Lecture 10: Inference (Spring 2025)
↳ Percy Liang explains why inference is memory limited.
↳ Then the fixes: quantization, speculative decoding and paged attention.

4) All About Transformer Inference (How To Scale Your Model, Google DeepMind)
↳ The math behind latency, throughput, memory and KV caches.
↳ Harder than the rest. Worth the extra effort.

All About Transformer Inference | How To Scale Your Model

Performing inference on a Transformer can be very different from training. Partly this is because inference adds a new factor to consider: latency. In this section, we will go all the way from sampling a single new token from a model to efficiently scaling a large Transformer across many slices of accelerators as part of an inference engine.

favicon jax-ml.github.io

5) MIT 6.5940 EfficientML (Fall 2024, Song Han)
↳ Watch the quantization lectures and Lecture 13 on LLM deployment.
↳ Slides and videos sit on the course page.

6) Inside vLLM (Aleksa Gordić)
↳ Scheduling, paged attention, prefix caching, speculative decoding, multi-GPU serving.
↳ Based on the vLLM V1 engine at one August 2025 commit, so some code has moved since.

7) Mini-SGLang (SGLang project)
↳ A compact serving engine with radix cache, chunked prefill and overlap scheduling.
↳ Reading the code is free. Running it needs Linux and an NVIDIA GPU with CUDA.

GitHub logo sgl-project / mini-sglang

A compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems.

Mini-SGLang

A lightweight yet high-performance inference framework for Large Language Models.

Mini-SGLang is a compact implementation of SGLang, designed to demystify the complexities of modern LLM serving systems. With a compact codebase of ~5,000 lines of Python, it serves as both a capable inference engine and a transparent reference for researchers and developers.

✨ Key Features

  • High Performance: Achieves state-of-the-art throughput and latency with advanced optimizations.
  • Lightweight & Readable: A clean, modular, and fully type-annotated codebase that is easy to understand and modify.
  • Advanced Optimizations
    • Radix Cache: Reuses KV cache for shared prefixes across requests.
    • Chunked Prefill: Reduces peak memory usage for long-context serving.
    • Overlap Scheduling: Hides CPU scheduling overhead with GPU computation.
    • Tensor Parallelism: Scales inference across multiple GPUs.
    • Optimized Kernels: Integrates FlashAttention and FlashInfer for maximum efficiency.
    • ...

🚀 Quick Start

⚠️ Platform Support: Mini-SGLang currently supports…





8) GPU MODE Lectures
↳ Start with Lectures 1, 3 and 4: profiling, getting started with CUDA, compute and memory architecture.
↳ The repo has the code and slides. The recordings are on their YouTube channel.

GitHub logo gpu-mode / lectures

Material for gpu-mode lectures

Supplementary Material for Lectures

YouTube Channel

The PMPP Book: Programming Massively Parallel Processors: A Hands-on Approach (Amazon link)

Lecture 1: Profiling and Integrating CUDA kernels in PyTorch

Lecture 2: Recap Ch. 1-3 from the PMPP book

Lecture 3: Getting Started With CUDA

Lecture 4: Intro to Compute and Memory Architecture

Lecture 5: Going Further with CUDA for Python Programmers

Lecture 6: Optimizing PyTorch Optimizers

Lecture 7: Advanced Quantization

Lecture 8: CUDA Performance Checklist

9) Triton Tutorials (official docs)
↳ Go from vector addition to fused softmax, matrix multiplication and fused attention.
↳ Each tutorial is runnable code you can download.

10) AIPerf (NVIDIA)
↳ Measure time to first token, inter token latency, throughput and goodput.
↳ Its guides cover trace replay, so you can test with real traffic patterns.

GitHub logo ai-dynamo / aiperf

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution.

AIPerf

PyPI version License Codecov Discord Ask DeepWiki

AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution. It provides detailed metrics using a command line display as well as extensive benchmark performance reports.

AIPerf UI Dashboard

Quick Start

This quick start guide leverages Ollama via Docker Desktop.

Setting up a Local Server

In order to set up an Ollama server, run granite4:350m using the following commands:

docker run -d \ --name ollama \ -p 11434:11434 \ -v ollama-data:/root/.ollama \ ollama/ollama:latest docker exec -it ollama ollama pull granite4:350m

Basic Usage

Create a virtual environment and install AIPerf:

python3 -m venv venv source venv/bin/activate pip install aiperf

Note

The optional crick dependency has no prebuilt wheel for some platforms including Linux aarch64 (arm64), and falls back to an sdist that needs a C compiler at install time. crick backs the non-default tdigest list-metric backend, so a plain pip…

Basic Python, PyTorch and Transformer knowledge will help.

All 10 are free to read or watch. Running the code examples may need a paid GPU.