The book
Rebuild a deep learning framework from a flat buffer of floats to GPT-2, one layer per chapter.
This book explains how kael works and how to rebuild it. It starts with a flat block of memory and a list of strides and ends with a GPT-2 checkpoint that loads, tokenizes text and generates it. Each chapter adds one layer on top of the previous ones, walks through the real code and ends with steps to write that layer yourself.
Who it is for
Engineers who write C++ and want to understand what a deep learning framework does internally. The book assumes C++20 and basic linear algebra: vectors, matrices and matrix multiplication. It does not assume prior knowledge of PyTorch internals, automatic differentiation, CUDA or transformers. Each of those is introduced from first principles.
How to read it
Read the chapters in order. Each one depends on the layers before it.
Most chapters match a git tag in the library repository. Open the tag in a separate worktree to see the code exactly as it was when that chapter was complete, and read the chapter next to it:
git clone https://github.com/JetBerri/kael.cpp
cd kael.cpp
git worktree add ../kael-views chapter-04-views
cmake -S ../kael-views -B ../kael-views/build
cmake --build ../kael-views/build
ctest --test-dir ../kael-views/buildTags are numbered by project phase, so chapter-02-shapes holds the code for chapter 01 and chapter-03-core the code for chapter 02. Chapter 00 has no tag. Chapter 12 starts at chapter-12-transformer and is complete at chapter-13-polish, which also marks v1.0.0.
How each chapter is organized
| Section | What it gives you |
|---|---|
| Why this layer exists | The problem the layer solves and a mental model to hold on to |
| Concepts | The idea from first principles, with worked numeric examples and diagrams |
| Implementation | The real types and functions, in the order you would write them |
| Edge cases and pitfalls | Failure modes the code guards against, and how |
| How it is tested | The test suites and golden data that prove the layer correct |
| Build it yourself | Ordered steps to reimplement the layer, each with a checkpoint |
| Check your understanding | Short questions with hidden answers |
Conventions
- Paths are relative to the root of the library repository.
- Code excerpts are trimmed for reading and name the file they come from. When an excerpt and the code disagree, the code wins.
- Shapes are written
[2, 3]and strides(3, 1).numelis the number of elements.
Chapters
- 00 Overview What kael is, how its layers depend on each other, where each piece lives in the repository, and how to build and test it.
- 01 Shapes and strides The index arithmetic that turns a flat buffer into an n-dimensional array: element counts, strides, offsets, broadcasting and dimension wrapping.
- 02 Memory model How kael turns shape arithmetic into real tensors: element types, storage, TensorImpl, the Tensor handle, factories, random numbers, printing and errors.
- 03 Views How kael reshapes, transposes, slices and broadcasts tensors by rewriting metadata over shared storage, and how it copies and writes safely when layouts alias.
- 04 TensorIterator The loop engine behind every elementwise op: type promotion, dtype dispatch, broadcasting with byte strides, dimension coalescing and a parallel two-level loop.
- 05 Operations Arithmetic, math functions, in-place ops, reductions, matmul, indexing, sequence ops and the softmax family, checked case by case against PyTorch.
- 06 Autograd Reverse-mode automatic differentiation over a dynamic graph: how kael records every operation and walks the graph backward to compute gradients.
- 07 Neural networks and training Modules, layers, initialization, losses, optimizers, schedulers, data loading and an MNIST classifier that trains end to end on a CPU.
- 08 CPU performance A thread pool, a six-step matmul ladder, a caching allocator and fused row kernels that make kael fast on a CPU without changing a single result.
- 09 CUDA backend Device memory, device kernels shared between a CUDA backend and a host emulation, and the rules that keep tensors on one device.
- 10 Python bindings A nanobind extension and a pure-Python layer that give kael a PyTorch-style API, zero-copy NumPy interop and GIL-free compute.
- 11 Transformer Causal self-attention, a GPT-2 style model with tied weights, a key/value cache and a sampling loop, built from the modules of chapter 07.
- 12 Loading GPT-2 The safetensors format with validated, atomic and memory-mapped I/O, the GPT-2 weight mapping, and a byte-level BPE tokenizer that matches tiktoken.
- 13 Serving and tooling An HTTP server for GPT-2 generation, debugging helpers for autograd, fuzzing, sanitizers, static analysis and the CI that runs all of it.