Building a language model from scratch on the திருக்குறள்

Play the model's game yourself

Below is the start of a real kural, cut off mid-verse. Your job is the LLM's job: guess the next character. Notice what you use to decide — letter patterns, grammar, meaning. Training a model means making it acquire those same instincts, purely from data.

The journey we're on

1. Corpus — curate the text the model will learn from. Done, explained below.
2. Tokenizer — turn text into numbers (and discover why Tamil makes this fun). Done, explained below.
3. Model — build a decoder-only transformer (the GPT architecture) in PyTorch.
4. Training — the loop: predict, measure error (loss), nudge weights, repeat.
5. Generation — sample from the trained model: temperature, top-k, and creativity.

Explore the raw material

This is the actual training text. The "model view" is exactly the string the model sees — no chapter titles, no numbers, no translations. Just verse, and a blank line between verses. Any structure the model ends up knowing, it learned from patterns in this stream.

The character inventory — our future vocabulary

Every distinct Unicode codepoint in the corpus. There are only — this becomes the char-level tokenizer's entire vocabulary in Stage 1. The single most common "character" is ் (puḷḷi, the dot that silences a consonant's inherent vowel) — a hint that Tamil text in Unicode is built from pieces smaller than what your eye reads as one letter.

One glyph ≠ one codepoint

What reads as one Tamil letter is often two codepoints: base consonant + combining vowel sign. கு (ku) = க (ka) +  ு (vowel sign u). Our char-level model predicts codepoints, so it must learn Tamil orthography — which combinations are legal — from scratch. Type anything Tamil to see its decomposition:

Train / validation split — the honesty mechanism

Our corpus is so small the model could eventually memorize it. A model that recites is not a model that learned. So we hide the 10th kural of every chapter (133 verses, exactly 10%) from training. If the model gets good at predicting those, it learned the style; if it only gets good on training verses, it's just memorizing. Watching these two numbers diverge during Stage 3 will be one of the best lessons of this whole project.

The design space

• Character-level — one token per codepoint. Tiny vocabulary ( here), nothing is ever out-of-vocabulary, but sequences are long and the model wastes capacity re-learning spelling.
• Word-level — one token per word. Huge vocabulary, and any unseen word breaks it. Essentially abandoned.
• Subword (BPE) — the middle path used by GPT, Llama, Claude: frequent strings become single tokens, rare strings decompose into pieces. We built it from scratch.

Tokenizer playground

Type any Tamil text (or pick a kural) and see how each tokenizer slices it. Alternating colors mark token boundaries; the small number under each chip is the token's integer id — the actual input to the neural network.

Watch BPE learn — merge by merge

BPE training is greedy compression: count every adjacent pair of tokens in the corpus, fuse the most frequent pair into a new token, repeat. Drag the slider to replay training. Watch the sample verse fuse from isolated codepoints into syllables (ன், ும்) then word-parts (ெல்லா) — nobody told it Tamil has syllables; frequency statistics discovered them.

The data's journey, top to bottom

The causal mask — why the model can't cheat

Training asks the model to predict character t+1 at every position t simultaneously, in one forward pass. That only works if position t provably can't see the future. One triangular mask does it: attention scores to later positions are set to −∞ before the softmax, which turns them into exactly zero weight. Hover any character:

Where do 815,744 parameters live?

Our model's exact size, computed live from the same formulas as the PyTorch code. Drag the sliders — watch where parameters go as you scale. (Head count is absent on purpose: heads split the same matrices, they don't add parameters.)

Our first real bug: initialization

Before training, a correctly-wired model should be exactly as good as uniform guessing: cross-entropy = ln(47) = 3.85. A number we can predict before running — a free correctness test. Our first run printed 81.4.

The cause: PyTorch initializes embeddings from N(0,1), and since our output head is tied to the embedding matrix, initial logits came out ~10 standard deviations wide. Softmax saturated; the model was born pathologically overconfident, and its "poetry" was அஅஅஅஅஅ… forever. The fix (GPT-2's): initialize every weight with std 0.02. After that, loss = 3.874 ≈ ln(47), and the untrained babble is honest uniform noise:

அீஉமரஞிபழீீஏ யநஊேீஉ
ூஎநைூ ஆவூைஉமஐத.நரபஃஐொஎூசவஆறஉயசஒநஈவாைஙுதழொ

Note the illegal Tamil — vowel signs after vowels (அீ), floating combining marks. This is our baseline. The first visible sign of learning in Stage 3 will be the model discovering orthography and refusing to write these.

The whole loop is five lines

for step in range(3000):
    x, y = get_batch()          # 64 random 128-char windows; y = x shifted by one
    logits, loss = model(x, y)  # cross-entropy: -ln P(true next char), averaged
    loss.backward()             # backprop: a gradient for every weight
    optimizer.step()            # AdamW nudges each weight against its gradient

The refinements around it (in kuralgpt/stage3_train.py) are all stability engineering: learning-rate warmup (small steps while weights are still random) then cosine decay, gradient clipping (cap any single step's size), and weight decay + dropout — our defences against memorising a 75 KB corpus.

One number to internalize: e^loss is the "perplexity" — the effective number of characters the model is choosing between. Untrained: e^3.85 = 47, i.e. clueless. Watch it collapse below.

The loss curves — and the moment memorisation begins

Both curves start at 3.85 (uniform guessing). Train loss is measured on verses the model studies; val loss on the 133 kurals it has never seen. Watch the gap: the curves fall together at first (everything learned is general), then train keeps dropping while val flattens — that growing wedge is memorisation, capacity spent photocopying the textbook instead of learning the subject. Our dropout and weight decay are why the wedge grows slowly instead of exploding.

Watch it learn to write

At milestones during training we froze the model and asked it to write (temperature 0.8, seeded with a newline — "start a fresh verse"). Drag through training time:

Now train one yourself — live, in this tab

Everything this chapter described, happening in front of you: a smaller sibling of our model (2 layers, 64-wide, 107,200 parameters) trains on the same 1,197 kurals — and this time backpropagation is hand-written. PyTorch computed our gradients in Stage 3; here every backward formula — through the softmax, the attention weights, the layernorms — is spelled out in site/train.js and gradient-checked against numerical derivatives. PyTorch is nowhere in the room.

— press Start, then watch this box every ~25 steps —

Single-threaded JavaScript in a Web Worker manages ~1 step/second — your GPU did ~30/second in Stage 3, on a model 8× larger. Same math, honest speed difference. A few minutes gets the loss from 3.85 (clueless) to ~2: watch spaces, then word-endings, then verse shape emerge in the samples.

The live poet — the trained model, running in this tab

Nothing below is a recording. The best checkpoint's 815,744 weights were exported to weights_char.bin (3.3 MB) and the forward pass — attention, KV cache and all — is ~200 lines of plain JavaScript (site/kural-engine.js), verified to reproduce PyTorch's outputs to within 0.00005. No server, no GPU, no libraries: your browser is doing the inference.

The sampling lab — real logits, live knobs

These are the actual output scores from our best checkpoint (step 2,650) for a few contexts. Your browser re-applies temperature (divide scores by t, then softmax) and top-k (keep only the k best, renormalise) in real time — the same math the generator runs:

Same model, same seed — different knobs

Every sample below starts from the identical random seed; only the decoding strategy differs. The text differences are 100% knob.

Look inside: what attention actually attends to

These are the real attention weights from inside the trained model, reading kural #1. Pick a layer and head, then hover any character: the shading on earlier characters shows how much this head consults each of them at that moment. (Rows always sum to 1 — attention is a budget.)

The anthology — six verses that never existed

Complete verses sampled at temperature 0.75 from the best checkpoint. Not one of these appears in the Thirukkural — every line is invented, in Valluvar's form and voice:

But do they mean anything? An honest translation

The ultimate test: translate them. Verdict — no. Every verse is built from real Tamil morphemes sitting in grammatically correct slots, yet none composes into a proposition. Where they fail is as instructive as where they succeed:

For contrast, a real kural (#129) — an argument compressed into seven metrical feet:

What you built — and the road from here to ChatGPT

End to end: 1,330 ancient verses → a 47-character vocabulary → an 815,744-parameter transformer → 102 seconds of gradient descent → a machine that composes passable kural form. What it does not have is meaning: it has never seen a word's definition, only character statistics. Everything it "knows" is pattern.

• Scale is the road from here to fluency: more data, wider and deeper models, BPE tokens instead of characters — same five-line training loop.
• Instruction tuning then teaches a fluent model to follow requests (fine-tune on prompt→response pairs), and RLHF/RLAIF tunes it toward answers people prefer. That's the full distance from KuralGPT to a chat assistant — and you now know every load-bearing idea in it.