Nano-LLM-from-scratch
language model · from scratch · Jul 2026 · Author
A small decoder-only transformer and byte-level BPE tokenizer written in PyTorch, with RoPE, RMSNorm, SwiGLU and a KV cache.
Result
4.94 to 2.90
validation loss over 2,500 CPU training steps
Written with RoPE, RMSNorm, SwiGLU, a KV cache and a byte-level BPE tokenizer.
257 offline tests
Results
| Measurement | Value |
|---|---|
| Training steps | 2,500 |
| Validation loss | 4.94 to 2.90 |
| Tests | 257 |
Stack
- Python
- PyTorch
- pytest