Fast-training LM experiments

A time-boxed experiment with one constraint: train a language model on a 2.2 GB corpus (about 588M tokens) in thirty minutes or less on an M3 Max, exploring non-traditional training methods rather than scaling down a standard recipe.

The result

The winning design was a two-system hybrid, not a single model. The first component is a counting LM — an order-5 n-gram model with interpolated absolute discounting, all keys stored as u64s.

That result is the interesting part. Under a hard wall-clock budget, a well-implemented counting model earns its place instead of being the thing you replaced with a neural network in the first chapter — because it trains at the speed of a single pass over the data, and the thirty minutes are then available to spend where they buy more.

Why it's archived

It answered its question. The findings fed directly into the architecture decisions for local-ai, which is where the ongoing work lives.