introduction

kael is a deep learning framework written from zero in modern C++. It has tensors with shapes and strides, views that share memory, a loop engine with broadcasting and type promotion, reverse-mode autograd, neural network modules and optimizers, a multithreaded CPU backend, a CUDA backend, Python bindings, and a GPT-2 that loads real weights and generates text.

Every layer is implemented from scratch, with no wrappers around another framework. Each one is tested against PyTorch and explained in a book that rebuilds it step by step.

#include <iostream>

#include "kael/kael.h"

int main() {
    kael::Tensor x = kael::randn({64, 784});
    kael::Tensor w = kael::randn({784, 10}).requires_grad_();

    kael::Tensor loss = kael::relu(kael::matmul(x, w)).sum();
    loss.backward();

    std::cout << *w.grad() << "\n";   // dloss/dw, shape=[784, 10]
}

results

Measured on an Apple M5 Pro (15 cores) for the CPU and on an NVIDIA Tesla T4 for the GPU. Every number comes from a command in the repository.

Workload	CPU, Apple M5 Pro	GPU, NVIDIA T4
MNIST MLP, 5 epochs	97.42% test accuracy, 0.62 s per epoch	97.18% test accuracy, 1.52 s per epoch
matmul, f32	557 GFLOP/s at 1024 x 1024	427 GFLOP/s at 2048 x 2048
GPT-2 small, text generation	15 tokens/s	14 tokens/s

Correctness is checked against PyTorch: 783 golden cases over 67 operations, generated by PyTorch and replayed by the C++ tests, plus 30 Python tests that compare kael with PyTorch and with the Hugging Face GPT-2 logits.

The GPU numbers are not higher than the CPU numbers. The MNIST model is too small to keep a GPU busy, and the CUDA kernels favor clarity over speed, as chapter 09 explains.

1. v1 naive i, j, k loops1.8
2. v2 i, k, j loop order35
3. v3 cache tiling36
4. v4 packed panels37
5. v5 NEON micro-kernel, one core124
6. v6 threaded, 15 threads557
7. PyTorch 2.15, Apple Accelerate1820

The naive loop is 300 times slower than v6 and 69 times slower than v5 on a single core. v6 reaches 31% of PyTorch, which calls Apple's vendor BLAS. Chapter 08 builds every step.

motivation

• Why does reshape sometimes copy and sometimes not?
• How does a + b work when a is [3, 1] and b is [4]?
• What does loss.backward() actually walk, and in which order?
• Why is a naive matrix multiply 300 times slower than a tuned one on the same CPU?
• What happens between a .safetensors file on disk and a sentence of generated text?

Frameworks such as PyTorch answer these questions in millions of lines of code. kael answers them in a codebase small enough to read in a few weeks, with every decision written down.

quick start

kael needs CMake 3.24 or newer and a C++20 compiler.

git clone https://github.com/JetBerri/kael.cpp
cd kael.cpp
cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure

Then start with the overview.

the book

Fourteen chapters. Each one adds a layer, shows the real code and ends with steps to write that layer yourself.

1. 00 Overview The architecture, the repository map and the build
2. 01 Shapes and strides The index arithmetic that turns a flat buffer into an n-dimensional array, and broadcasting
3. 02 Memory model Data types, storage, TensorImpl, the Tensor handle, factories and random numbers
4. 03 Views Reshape, transpose, slicing and expand without copying data
5. 04 TensorIterator The loop engine behind every elementwise op: broadcasting, type promotion and dimension coalescing
6. 05 Operations Arithmetic, math functions, reductions, matmul, indexing and softmax, checked against PyTorch
7. 06 Autograd Reverse-mode automatic differentiation over a dynamic graph
8. 07 Neural networks and training Modules, losses, optimizers, data loading and an MNIST classifier
9. 08 CPU performance A thread pool, a fast matmul, a caching allocator and fused kernels
10. 09 CUDA backend Device memory, device kernels and a host emulation backend for testing
11. 10 Python bindings A Python package on top of the C++ library, with NumPy interop
12. 11 Transformer Attention, a GPT model, a KV cache and sampling
13. 12 Loading GPT-2 The safetensors format, the GPT-2 weight mapping and the BPE tokenizer
14. 13 Serving and tooling An HTTP server, debugging tools, fuzzing, sanitizers and CI

a growing knowledge base

The book explains how kael works. The concepts section explains the ideas around it: the math behind gradients, how memory hierarchies shape kernels, why attention works. It is open to contributions and meant to grow with the community.

Every page on this site is a Markdown file in a public repository. Fix a sentence, add an exercise, or write the article you wish you had read when you started. Read how to contribute, or request a topic you want explained.