A small language model that reads like a lawyer.
slm125m is a 125.8-million-parameter transformer pretrained from nothing on US court opinions, SEC filings and educational web text. No fine-tuning of an existing model — the corpus, the tokenizer and every weight were built from scratch for $31.76 of compute. It was then instruction-tuned on a synthetic Q&A set built from its own corpus, so it answers questions from a passage — and says so when the passage doesn't contain the answer.
By the numbers
Every figure below is measured, not estimated — from the training logs and the final held-out evaluation.
How it was built
Four CPU stages produced the corpus; one GPU run produced the model.
Clean & deduplicate
718,780 documents streamed from US case law, SEC filings and FineWeb-Edu, then put through a six-step deterministic filter — boilerplate, repetition, language and an OCR-garble gate for scanned court records. 97.1% survived.
Decontaminate
MinHash/LSH removed near-duplicates, and a 13-gram overlap check against the CaseHOLD benchmark stripped 24,002 contaminated documents so the model is never scored on text it memorised.
A vocabulary of its own
A 16,384-token byte-level BPE trained on this corpus rather than borrowed. Fitted to legal register, it encodes the same text in ~5% fewer tokens than a generic vocabulary would.
8×H100, 47 minutes
15,789 steps at a 524,288-token global batch, bf16 with a cosine schedule annealed to its floor. Validation perplexity fell 15.48 → 8.11 and never once regressed.
Playground
Running live on the actual weights — no cached responses.