Building a language model from scratch on the திருக்குறள்

Play the model's game yourself

Below is the start of a real kural, cut off mid-verse. Your job is the LLM's job: guess the next character. Notice what you use to decide — letter patterns, grammar, meaning. Training a model means making it acquire those same instincts, purely from data.

The journey we're on

1. Corpus — curate the text the model will learn from. Done, explained below.
2. Tokenizer — turn text into numbers (and discover why Tamil makes this fun). Done, explained below.
3. Model — build a decoder-only transformer (the GPT architecture) in PyTorch.
4. Training — the loop: predict, measure error (loss), nudge weights, repeat.
5. Generation — sample from the trained model: temperature, top-k, and creativity.

Explore the raw material

This is the actual training text. The "model view" is exactly the string the model sees — no chapter titles, no numbers, no translations. Just verse, and a blank line between verses. Any structure the model ends up knowing, it learned from patterns in this stream.

The character inventory — our future vocabulary

Every distinct Unicode codepoint in the corpus. There are only — and this short list matters more than it looks: in the next chapter it becomes the complete set of symbols the model reads and writes with — its entire alphabet of possible guesses. The single most common "character" is ் (puḷḷi, the dot that silences a consonant's inherent vowel) — a hint that Tamil text in Unicode is built from pieces smaller than what your eye reads as one letter.

One glyph ≠ one codepoint

What reads as one Tamil letter is often two codepoints: base consonant + combining vowel sign. கு (ku) = க (ka) +  ு (vowel sign u). Our model will read and guess codepoints, one at a time — so it must learn Tamil orthography — which combinations are legal — from scratch. Type anything Tamil to see its decomposition:

Why numbers at all?

Because a neural network isn't a text machine — it's an arithmetic machine. Strip the mystique and the entire transformer is multiply-and-add: vectors times matrices, millions of times. There is no operation in it that can act on the letter க; there are only operations that act on numbers. So before anything else, every piece of text must become a sequence of integers — and at the far end, the model's numeric output must map back to text. The tokenizer is that bridge, in both directions, and it must be perfectly reversible: decode(encode(text)) == text.

One subtlety worth flagging now: the integer itself carries no meaning. க → 16 is an arbitrary label, like a locker number — id 16 and id 17 are not "close". Giving those labels meaning is a separate, learned step called an embedding, and it's the first thing the model does in Chapter 3.

The same three choices, in engineering terms

• Character-level — one token per codepoint. Tiny vocabulary ( here), nothing is ever out-of-vocabulary, but sequences are long and the model wastes capacity re-learning spelling.
• Word-level — one token per word. Huge vocabulary, and any unseen word breaks it. Essentially abandoned.
• Subword (BPE) — the middle path used by GPT, Llama, Claude: frequent strings become single tokens, rare strings decompose into pieces. We built it from scratch.

Tokenizer playground

Type any Tamil text (or pick a kural) and see how each tokenizer slices it. Alternating colors mark token boundaries; the small number under each chip is the token's integer id — the actual input to the neural network.

Watch BPE learn — merge by merge

BPE training is greedy compression: count every adjacent pair of tokens in the corpus, fuse the most frequent pair into a new token, repeat. Drag the slider to replay training. Watch the sample verse fuse from isolated codepoints into syllables (ன், ும்) then word-parts (ெல்லா) — nobody told it Tamil has syllables; frequency statistics discovered them.

Why is an embedding a vector, and not just a number?

A software engineer's first instinct: the tokenizer already gave each character an integer — why not feed that in? Two reasons. First, the ids are arbitrary labels: nothing about 16 vs 17 says க and ங behave alike — but arithmetic on raw ids would bake in exactly that false claim (17 is "close to" 16, "twice" 8.5…). The net would waste its capacity fighting its own input encoding.

Second — and this is the deeper one — one number is one axis of similarity. With a single number per character, you can express only one way for characters to resemble each other (their position on that one line). But characters are similar in many independent ways at once: vowel-ness, how often they end a word, what they do to the following consonant, whether they can start a verse. Each of those needs its own dimension. A 128-number vector gives the model 128 independent properties to describe a character with — and it gets to invent what the properties mean, because every one of those 6,016 table entries (47 × 128) is a trainable weight, nudged by every training step.

Similarity then stops being a lookup and becomes geometry: characters that behave alike drift to nearby points, because predicting well forces them to. You can check that claim on the real model below.

The map the model drew of its own alphabet

These are the trained embedding vectors from our checkpoint — the actual 47 rows of tok_emb.weight — flattened from 128 dimensions to the 2 that capture the most variance (PCA). Nobody labelled anything: any structure you see, the model invented from prediction pressure alone. Click a character to see its true nearest neighbours measured in the full 128-dimensional space:

↑ cosine similarity: 1.0 = same direction ("used identically"), 0 = unrelated, negative = used in opposing contexts.

Dot product → softmax → weights: the mechanism, with your hands on it

The classroom story above becomes concrete arithmetic in three moves. (1) A match score between a query and a key is their dot product: multiply the vectors element-by-element and sum. Geometrically that measures alignment — vectors pointing the same way score high, perpendicular ones score ~0, opposite ones score negative. "How aligned is what I seek with what you contain" becomes one number. (2) Raw scores can be anything (−7.3, 0.2, 4.1…), but we need mixing proportions — positive, summing to 1. Softmax does that: exponentiate each score (now all positive, big gaps amplified), divide by the total. (3) The output is the values blended in those proportions. That's all attention is.

Below, one query (turmeric) and two keys in a 2-D toy space. Drag the sliders to rotate the keys and watch the dot products — and the two tokens' share of the attention budget — respond:

Multi-head: why four heads don't learn the same thing four times

Our model splits its 128-dimensional workspace into 4 heads of 32 dimensions, each with its own slice of the Q, K and V matrices, each running the mechanism above independently. The obvious worry: they train together, on the same text, toward the same goal — what stops all four converging on the same (best) trick?

Random initialization breaks the tie, and training keeps it broken. At birth the four heads hold different random weights, so each starts out accidentally slightly better at different patterns. Now follow the incentives: the model's guesses are scored on the combined contribution of all four heads. If head 3 already supplies "which character came right before me", a second head supplying the same information adds nothing — the guesses that information fixes are already good, so there is little pressure on it to keep doing that. The strong pressure comes from mistakes nobody is covering yet. Each head is pushed hardest toward the work left over by the others. Specialisation isn't designed in; it's the stable equilibrium of four learners sharing one pay-check.

This is the same reason your colleagues on a shared codebase drift into specialities. And it's testable: in Chapter 5's attention explorer you'll find, in the trained weights, a previous-character head, a start-of-word head, and a head that reads line 1 while writing line 2 — jobs nobody assigned.

The causal mask — why the model can't cheat

Training asks the model to predict character t+1 at every position t simultaneously, in one pass through the model. That only works if position t provably can't see the future. One triangular mask does it: attention scores to later positions are set to −∞ before the softmax, which turns them into exactly zero weight. Hover any character:

Why stack layers — and what the different layers end up doing

One attention-plus-MLP block is useful but shallow: attention there can only combine raw embeddings — "this character, near that character". The power move is feeding its output into another identical block. Now layer 2's attention operates on vectors that are already mixtures — a position no longer means "the character த" but "a த that follows மு, mid-word, in line 2" — so layer 2 can express patterns about patterns. Every extra layer is one more round of composition, exactly like building software: functions of functions, each level speaking a slightly richer vocabulary than the one below.

Because of that, depth creates a natural division of labour that nobody programs in. In our 4-layer model you can watch it in Chapter 5's attention explorer: early layers hug local context (the previous character — spelling-level work), while a deeper layer grows the head that consults line 1's opening while writing line 2 — verse-level structure that only makes sense once the low-level features exist. Interpretability work finds the same gradient in frontier models, stretched over ~100 layers: early layers resolve tokens and syntax, middle layers assemble meanings and facts, late layers shape the actual next-token decision. Same staircase, more steps.

Where do 815,744 parameters live?

Our model's exact size, computed live from the same formulas as the PyTorch code. Drag the sliders — watch where parameters go as you scale. (Head count is absent on purpose: heads split the same matrices, they don't add parameters.)

The MLP's bend: why ReLU, GELU and SwiGLU exist

The MLP is two matrix multiplications with a curve squeezed between them — and the curve is the entire point. Here's the trap it avoids: a stack of linear layers collapses into one linear layer. If the MLP were just W₂(W₁·x), that equals (W₂W₁)·x — a single matrix. Stack our whole model 4 blocks deep without nonlinearities and 815,744 parameters would have exactly the expressive power of one matrix multiply: straight lines only, no "if-then" behaviour, no feature that switches on in one context and off in another. The nonlinearity is what lets depth mean something.

The curves themselves, oldest to newest:

• ReLU — max(0, x). Brutally simple: negative inputs are silenced, positive pass through. Each of our 512 hidden units becomes a learned feature detector that is either off or proportionally on. Weakness: a unit stuck on the flat side gets zero gradient — it can die and never recover.
• GELU (ours, and GPT-2/3's) — a smoothed ReLU: instead of a hard gate it multiplies x by "the probability this input should pass". No dead zone, gradient everywhere, slightly better training. Same job, rounded corners.
• SwiGLU (Llama, Mistral, most 2026 models) — a different shape entirely: the layer computes two projections of x and multiplies them — silu(W₁x) ⊙ W₂x — so one half acts as a learned gate on the other. Multiplying two functions of the input lets the layer express interactions ("pass feature A through only when feature B is present") that a fixed curve can't. Costs a third matrix; consistently wins at scale.

Keep the proportions in mind from the chart above: this "expand 4×, bend, contract" unit is where ~⅔ of all parameters live, in our model and in frontier models alike. Attention decides what to look at; this is where the model thinks about it — and, at scale, where most factual knowledge is stored.

The data's journey, top to bottom

Every piece is now on the table — embeddings, attention, the causal mask, heads, the MLP. Here is the whole machine assembled: follow one batch of token ids from input to prediction.

LayerNorm — the unglamorous part that makes depth trainable

Between the famous parts, our forward pass keeps calling norm(x). What it does is almost insultingly simple: for one token's 128-number vector, subtract the mean, divide by the standard deviation — the vector is now centred with spread 1 — then let the model re-scale and re-shift with two small learned vectors (γ, β). No mixing between tokens, no mixing between batch entries: each token's vector is normalised alone.

Why bother? Remember the residual stream: every attention head and MLP adds its note onto the running vector. Adding never shrinks things, so by block 3 the stream's magnitude is whatever history made it — and everything downstream is scale-sensitive. Attention scores get sharper as vectors grow (the init bug in the next card is a spectacular demo of softmax saturating); training updates scale the same way, so drift compounds: a slightly-too-big layer feeds a bigger one. LayerNorm is the reset valve — every sub-layer receives its input at a known, standard scale, no matter what the stream has accumulated. That's the difference between deep stacks training reliably and the exploding/vanishing chaos that made pre-2015 deep nets miserable.

Two footnotes that connect it forward: placement and diet. We put the norm before each sub-layer (x + f(norm(x)), "pre-norm") so the raw residual stream itself is never squashed — that's the modern arrangement every frontier model uses. And RMSNorm (Llama & friends) is LayerNorm on a diet: skip the mean-subtraction, just divide by the root-mean-square and re-scale. One less pass over the vector, indistinguishable quality — the kind of 5% savings that matters when you run the layer a trillion times.

Our first real bug: initialization

Before training, a correctly-wired model should be exactly as good as uniform guessing — no better, no worse. Chapter 4 defines the score we use (the "loss" — average surprise at the true next character; lower is better), but its value for a clueless 47-way guess can be computed on paper: ln(47) = 3.85. A number we can predict before running — a free correctness test. Our first run printed 81.4.

The cause: PyTorch initializes embeddings from N(0,1), and since our output head is tied to the embedding matrix, initial logits came out ~10 standard deviations wide. Softmax saturated; the model was born pathologically overconfident, and its "poetry" was அஅஅஅஅஅ… forever. The fix (GPT-2's): initialize every weight with std 0.02. After that, loss = 3.874 ≈ ln(47), and the untrained babble is honest uniform noise:

அீஉமரஞிபழீீஏ யநஊேீஉ
ூஎநைூ ஆவூைஉமஐத.நரபஃஐொஎூசவஆறஉயசஒநஈவாைஙுதழொ

Note the illegal Tamil — vowel signs after vowels (அீ), floating combining marks. This is our baseline. The first visible sign of learning in Stage 3 will be the model discovering orthography and refusing to write these.

The whole loop is five lines

for step in range(3000):
    x, y = get_batch()          # 64 random 128-char windows; y = x shifted by one
    logits, loss = model(x, y)  # cross-entropy: -ln P(true next char), averaged
    loss.backward()             # backprop: a gradient for every weight
    optimizer.step()            # AdamW nudges each weight against its gradient

The refinements around it (in kuralgpt/stage3_train.py) are all stability engineering: learning-rate warmup (small steps while weights are still random) then cosine decay, gradient clipping (cap any single step's size), and weight decay + dropout — our defences against memorising a 75 KB corpus.

One number to internalize: e^loss is the "perplexity" — the effective number of characters the model is choosing between. Untrained: e^3.85 = 47, i.e. clueless. Watch it collapse below.

Train / validation split — the honesty mechanism

Before trusting anything the loop above reports, one problem needs naming: our corpus is so small the model could eventually memorize it — and a model that recites is not a model that learned. The defence was built back in Stage 0, before a single gradient step: we hid the 10th kural of every chapter (133 verses, exactly 10%) from training. The model studies the 1,197 grey verses below and is quizzed on the blue ones it has never seen. If it gets good at predicting those, it learned the style; if it only gets good on training verses, it's just memorizing. Watching those two numbers diverge in the chart below is one of the best lessons of this whole project.

The loss curves — and the moment memorisation begins

Both curves start at 3.85 (uniform guessing). Train loss is measured on verses the model studies; val loss on the 133 kurals it has never seen. Watch the gap: the curves fall together at first (everything learned is general), then train keeps dropping while val flattens — that growing wedge is memorisation, capacity spent photocopying the textbook instead of learning the subject. Our dropout and weight decay are why the wedge grows slowly instead of exploding.

Watch it learn to write

At milestones during training we froze the model and asked it to write — always with the same settings (a touch of randomness in its choices; Chapter 5 dissects those knobs) and the same starting cue, a newline: "begin a fresh verse". So every difference you see below is purely what training changed. Drag through training time:

Now train one yourself — live, in this tab

Everything this chapter described, happening in front of you: a smaller sibling of our model (2 layers, 64-wide, 107,200 parameters) trains on the same 1,197 kurals — and this time backpropagation is hand-written. PyTorch computed our gradients in Stage 3; here every backward formula — through the softmax, the attention weights, the layernorms — is spelled out in site/train.js and gradient-checked against numerical derivatives. PyTorch is nowhere in the room.

— press Start, then watch this box every ~25 steps —

Single-threaded JavaScript in a Web Worker manages ~1 step/second — your GPU did ~30/second in Stage 3, on a model 8× larger. Same math, honest speed difference. A few minutes gets the loss from 3.85 (clueless) to ~2: watch spaces, then word-endings, then verse shape emerge in the samples.

The live poet — the trained model, running in this tab

Nothing below is a recording. The best checkpoint's 815,744 weights were exported to weights_char.bin (3.3 MB) and the full forward computation — attention, KV cache (next card) and all — is ~200 lines of plain JavaScript (site/kural-engine.js), verified to reproduce PyTorch's outputs to within 0.00005. No server, no GPU, no libraries: your browser is doing the inference.

The KV cache — the trick that makes the poet above affordable

Generation has an ugly cost structure if you're naive about it. To produce character 201, the model needs attention over characters 1–200 — so the obvious implementation re-runs the full pass through the model over all 200 every step. Character 202 re-runs 201. Generating n characters costs 1+2+…+n ≈ n²/2 token-passes: the poet above would get visibly slower as the verse grows.

The fix falls out of the causal mask. Position 57's key and value vectors depend only on positions ≤ 57 — and those never change while we generate. So computing them again every step produces bit-identical numbers. Instead: cache them. Each step, compute q, k, v for the one new token, append k and v to a per-layer cache, and attend over the cache. Cost per token becomes flat, and the poet streams at a constant rate (watch the chars/sec readout above — it doesn't decay).

The price is memory, and it's worth internalising the formula: layers × 2 (K and V) × context × width × bytes. For us that's 4 × 2 × 128 × 128 × 4 = 512 KB — nothing. For a frontier model with 80 layers, 8,192 width and a 100k-token conversation, the same formula is tens of gigabytes, per conversation — the KV cache, not the weights, is what long context actually costs. Chapter 6 shows the surgery (grouped-query attention) invented to shrink it.

The sampling lab — real logits, live knobs

These are the actual output scores — logits, one raw number per vocabulary character, the model's native output before anything is a probability — from our best checkpoint (step 2,650) for a few contexts. Your browser re-applies temperature (divide logits by t, then softmax), top-k (keep only the k best) and top-p in real time — the same math the generator runs:

Same model, same seed — different knobs

Every sample below starts from the identical random seed; only the decoding strategy differs. The text differences are 100% knob.

How does it know when to stop?

Honest answer for our model: it doesn't. The generate loop is while(true) — the model happily writes verse after verse until our code cuts it off at a character budget. But look closer and it did learn a soft version of stopping: in the corpus every verse ends with a period, a newline, and a blank line, so after a well-formed second line the probability mass piles onto exactly that .\n\n sequence. The model learned "a kural is over now" as just another next-character pattern. What it lacks is a way to say "and I have nothing further to add."

Production models close that gap with a trick you now have the tools to appreciate: invent one more token that means "the end" — an EOS (end-of-sequence) token like GPT's <|endoftext|>. During training, every document in the corpus gets it appended, so the model learns to predict it exactly where a text should genuinely finish. During generation it's sampled like any other token — and when it comes up, the serving code stops the loop. The model's sense of "done" is a learned prediction, same as everything else it knows. Chat assistants extend the idea: a second phase of training (post-training — Chapter 6) teaches an end-of-turn token, which is how the model yields the floor instead of continuing your sentence. On top of that sit mundane hard guards — stop-sequences and max-token limits — because a sampled token is a probabilistic promise, and servers prefer certainties.

Look inside: what attention actually attends to

These are the real attention weights from inside the trained model, reading kural #1. Pick a layer and head, then hover any character: the shading on earlier characters shows how much this head consults each of them at that moment. (Rows always sum to 1 — attention is a budget.)

The anthology — six verses that never existed

Complete verses sampled at temperature 0.75 from the best checkpoint. Not one of these appears in the Thirukkural — every line is invented, in Valluvar's form and voice:

But do they mean anything? An honest translation

The ultimate test: translate them. Verdict — no. Every verse is built from real Tamil morphemes sitting in grammatically correct slots, yet none composes into a proposition. Where they fail is as instructive as where they succeed:

For contrast, a real kural (#129) — an argument compressed into seven metrical feet:

Encoder–decoder, encoder-only, decoder-only: the family tree

The 2017 transformer paper wasn't trying to build a chatbot — it was a translation system, and translation shaped its anatomy: read the whole source sentence first, then write the target. That gave it two halves, and the field spent its next years discovering which halves it actually needed:

Why did decoder-only win? Three compounding reasons. Training data: next-token prediction needs no labels, no pairs, no masking scheme — every text ever written is training data as-is. Generality: any task you can state in text becomes next-token prediction — translation is just "French: … English:" and a continuation; the task distinction the encoder–decoder hard-wires into architecture, decoder-only handles in content. Simplicity at scale: one tower, one objective, uniform blocks — exactly what you want to replicate across ten thousand GPUs. The bitter-lesson pattern: the architecture with the least built-in structure and the most appetite for data wins.

RoPE — positions as rotations

Our model stores a learned vector per seat number: slot 0 has a vector, slot 1 has a vector… 128 slots, end of story. Two problems you can see from here. The table has a hard edge — there is no slot 129, so the context length is a birth defect. And it's absolute: the pattern "vowel sign follows its consonant" gets learned separately at slot 3, slot 40, slot 97 — the model can't say "one position apart, anywhere", which is almost always what language patterns mean.

RoPE (Rotary Position Embedding) fixes both with one geometric idea: stop adding position information, start rotating with it. Take each query and key vector, treat its 128 numbers as 64 two-dimensional pairs, and rotate each pair by an angle proportional to the token's position — like clock hands, each pair ticking at its own speed (fast hands for fine distinctions, slow hands for coarse ones). Now recall: attention scores are dot products, and a dot product depends only on the angle between two vectors. Rotate a query at position 57 and a key at position 52, and the angle between them encodes exactly 57 − 52 = 5 — the positions themselves cancel. Slide the pair anywhere in the context and the score is untouched:

Both tokens slide together; each is rotated by its own absolute position — yet the angle between them (and so their attention score) never changes. Relative position falls out of the geometry for free.

That's why RoPE models extend to contexts far beyond anything seen in training — a pattern learned as "5 apart" works at position 100,000 — and why every notable open model since Llama uses it. The learned table you built is the honest baseline that makes the elegance legible.

Grouped-query attention — shrinking the KV cache 8×

Chapter 5 left a bomb ticking: the KV cache formula layers × 2 × context × width × bytes reaches tens of gigabytes for a frontier model holding a long conversation — per conversation. The weights are shared by every user; the cache is not. Serving cost at long context is KV-cache cost.

Now the observation that saves the day. In multi-head attention every head carries three matrices — Q, K, V — and so its own cache. But the roles aren't symmetric: queries are the questions, keys/values are the library. Heads genuinely need different questions — that diversity is what Chapter 3's multi-head card was about. But do 64 heads need 64 separate libraries of the same context? Measured answer: no. Grouped-query attention (GQA) keeps all 64 query heads but has them share a handful of K/V heads — Llama-70B uses 8, so groups of 8 questioners read from one shared library shelf. Cache: 8× smaller. Quality: within noise of full multi-head. (The extreme version, one shared K/V for all heads — "multi-query" — was tried first and does measurably hurt; 8 groups is the sweet spot the field settled on.)

In our terms: KuralGPT's 4 heads each cache their own 32-wide K and V slices. A GQA version would keep 4 kinds of questions but one shared 32-wide K/V — same formula, C_kv instead of C, and that single substitution is worth a fortune at scale.

Dense vs. mixture-of-experts — paying for only the knowledge you use

In our model every character flows through every parameter — the MLP is dense. Scale that honestly and you hit an absurdity: to make the model know more, you widen the MLPs; but then every token — every "the", every comma — pays the full compute bill for all of it. Knowledge and per-token compute are locked together.

Mixture-of-experts (MoE) unlocks them. Replace each block's one big MLP with N smaller "expert" MLPs (Mixtral: 8) plus a router — a tiny learned layer that scores, per token, which experts should handle it — and send each token through only the top 2. The result reads like an accounting trick but is real: Mixtral stores 46.7B parameters of knowledge yet spends only ~12.9B parameters of compute per token. Most 2026 frontier models are MoE for exactly this reason.

Three honest footnotes. The "experts" are not human-legible specialists ("the chemistry expert") — routing turns out to key on token-level statistics, and interpretability there is still murky. The router is trained end-to-end with everything else, with an extra load-balancing loss so it can't collapse onto two favourite experts. And nothing is free: all 46.7B parameters must sit in memory ready to be chosen — MoE buys compute, not RAM. Note it's the MLPs that get this treatment, not attention: Chapter 3 told you the MLPs are where the parameters (and the knowledge) live, so they're the part worth rationing.

Speculative decoding — spending a small model to speed up a big one

Here's an asymmetry you've already met without naming it. Generating is serial: one forward pass per token, each waiting for the last. But checking is parallel — Chapter 3's causal mask meant training scores predictions at every position of a sequence in one forward pass. So: producing 5 tokens costs 5 big-model passes, but verifying 5 proposed tokens costs 1. That gap is exploitable.

Speculative decoding: keep a small, fast draft model (think KuralGPT-sized) alongside the big one. The draft cheaply proposes a run of, say, 5 tokens. The big model runs once over all 5, getting its own next-token distribution at every position, and accepts the proposals one by one until the first disagreement — a coin-flip correction scheme makes the accepted stream provably identical in distribution to the big model sampling alone. Easy text ("…de que" after "más", boilerplate, code syntax) gets drafted 5-at-a-time and rubber-stamped; hard text falls back to the big model's own choice. Typical speedup: 2–3× with zero quality change — pure win, bought with the observation that verification parallelises and generation doesn't.

Pre-training vs. post-training — why a next-token guesser answers your questions

Every gradient step on this page is pre-training: minimise next-token surprise on raw text. Scale that to trillions of tokens and you get a base model — staggeringly knowledgeable, and not an assistant. Ask a base model "What is the capital of France?" and a perfectly plausible continuation is another list of geography questions — that's what such text looks like on the internet. It has the knowledge; it has no reason to believe it's in a conversation. You've seen the microcosm: KuralGPT continues verse because verse is all its universe contains.

Post-training is the (comparatively tiny) second phase that turns the continuation engine into a conversational tool:

• Supervised fine-tuning (SFT): same training loop, new corpus — tens of thousands of curated conversations in a chat template. The base model learns that text shaped "user asks → assistant answers helpfully" is what it should be continuing. Mechanically identical to Chapter 4; only the data changed.
• Preference tuning (RLHF/RLAIF): generate multiple answers, have humans — or an AI judge steered by a constitution — pick the better one, and nudge the model toward preferred answers. This is where "loss on text" stops being the target and "answers people actually want" starts — tone, honesty, refusals, formatting.

Two demystifying corollaries. The chat interface is text all the way down: system prompt and turn markers are literally tokens in the context window, and the assistant's "personality" is a pattern post-training made likely — there is no second machine behind the curtain. And the division of labour is stark: post-training's thousands of conversations can't teach what pre-training's trillions of tokens didn't — capability comes from pre-training; post-training aims it.

Your model vs. a 2026 frontier model — the whole chapter on one table

The skeleton you built — embeddings → causal attention → MLP, stacked on a residual stream, trained on next-token loss — is the frontier architecture. Everything this chapter added is a component swap on that skeleton, each solving a pain you personally felt:

Everything else — the causal mask, the residual stream, the training loop, temperature and top-k — is identical to what's on this page. The one live challenge to the skeleton itself: state-space models (Mamba), which replace attention entirely with a learned running summary — the first credible fork in the road since 2017.

What you built — and the road from here to ChatGPT

End to end: 1,330 ancient verses → a 47-character vocabulary → an 815,744-parameter transformer → 102 seconds of gradient descent → a machine that composes passable kural form. What it does not have is meaning: it has never seen a word's definition, only character statistics. Everything it "knows" is pattern.

• Scale is the road from here to fluency: more data, wider and deeper models, BPE tokens instead of characters — same five-line training loop.
• Post-training (the card above) then aims the fluent model at conversation. That's the full distance from KuralGPT to a chat assistant — and you now know every load-bearing idea in it, down to the gradients.