Not in This Model
Techniques the code does not use, from the rows of the documentation contract confirmed absent.
Each technique below is absent from the code at the pinned commit. Some appear in the reference PDF as a contrast, a future upgrade or removed earlier work, and the PDF does not mention the others. None of them is something this project does.
This is the only page that describes them. Sources & Discrepancies may quote one where it reports what the PDF says. bun run verify:content fails if one of these names appears anywhere else.
The rows are the ones contract/PDF_VS_CODE.md marks as not in the code. Techniques the code has in part, or only in tests, are not listed here. Those are described on the pages for that code, such as Gradient Checking.
| Technique | In the reference PDF | Note |
|---|---|---|
| Post-LN | reference title only | |
| BatchNorm | no | |
| RMSNorm | no | |
| Dropout | no | |
| Label smoothing | no | |
| Sinusoidal positions | no | |
| Rotary positions | no | |
| Weight tying | yes, as not used | head_w is a separate matrix from tok_emb. |
| GELU | yes, as future upgrade | |
| Upstream epsilon loss | yes, contrast | |
| N(0, 0.02^2) embedding init | yes, contrast | |
| Lower-casing | yes | The code preserves case |
| Pre-token encoding cache | yes | |
| Pad, BOS, EOS tokens | no | |
| L2 penalty in the loss | yes, contrast | |
| Plain gradient descent | yes, motivation | |
| Beam search, repetition penalty | no | |
| KV cache | no | |
| Six-rung ladder of checks | yes | |
| Stop when validation rises | yes, advice | No early stopping in code |
ELIZA chat (chat.py) | yes, removed | |
XOR network (xor_net.py) | yes, removed | |
Skip-gram (skipgram.py) | yes, removed | |
| Instruction following | yes, out of scope |
For how the code differs from the reference PDF in other ways, see Sources & Discrepancies.