How an LLM works — let's learn by building one

Introduction

What are language models? A language model is something that generates language on the go — one word at a time. Ask it something, and it composes its answer word by word, each new word chosen based on everything that came before it.

You can think of our brain as working on similar lines. When we speak, we don't plan the whole sentence in advance — the next word just arrives, and then the next. We never spend time thinking about how; the brain does the magic. A language model is a machine built to play that same game: given the words so far, produce the next one. That is all a language model is — and the rest of this write-up is about building one.

For our build, the source text is the Thirukkural (திருக்குறள்) — a classical Tamil work of 1,330 verses, each just two lines, written some 2,000 years ago. That little book is the entire world our model will know: everything it learns, it learns from those 1,330 verses.

For our build, the source text is a little book of classical Japanese haiku in English translation — Japanese Haiku (tr. Peter Beilenson, 1955, public domain): 215 short poems by Basho, Issa, Buson and friends, each just three or four short lines. That little book is the entire world our model will know: everything it learns, it learns from those 215 poems.

Vocabulary

First we need to build a vocabulary. When I say I speak English, I understand its 26 characters, and whatever I generate from my mind is going to come out of those 26 characters. In the same way, we need to define the vocabulary we are going to teach this system to generate from.

If I do some programming and pull out the unique characters from the Thirukkural's 1,330 verses, it comes to about 47. (Each Tamil character needs its own unique code point — and what looks like one letter, கு, is actually two code points, க + ு. Every glyph is built this way.) We consider these 47 as our vocabulary.

If I do some programming and pull out the unique characters from these 215 haiku, it comes to 36 — the 26 lowercase letters, plus space, newline, and a handful of punctuation marks. We consider these 36 as our vocabulary.

So now we know the goal, and we know the raw material: we teach the system to read this text again and again, learn from it, and then ask it to generate. That's our goal. But should our vocabulary really be just the characters? Or whole words, or something in between? Let's settle that now.

What would be the easiest choice for this vocabulary? We could say all the words in Tamil or EnglishEnglish — but that is too many. For example, "go", "going", and "gone" all count as unique words. Tamil makes it even worse: from போ (go) you get போகிறேன் (I am going), போனான் (he went), போவாள் (she will go), போகமாட்டேன் (I will not go)… Tamil glues tense, person, and even negation into the word itself, so every form is a new "word". And any word the corpus never contained simply would not exist for the model. The vocabulary just explodes.

What about character by character? Then generation becomes expensive: instead of producing a word (or something in between) in one go, we produce one character at a time, so the cost is just too high.

What about something in the middle? Instead of keeping "going" as one word-token, we have "go" as a token and "ing" as a token. Whenever the language model wants to produce "going", it can generate two tokens — "go" plus "ing" — and it can still make up the word.

The thought process is that a token will not be a full word, and it will not be a single character. It will be something in between — base words, plus the endings — so they can be arranged to create other words. This idea is called subword tokenization, and the algorithm that discovers these pieces is called byte pair encoding (BPE). That is what real language models use.

For our project, though, we will keep it simple and honest: our tokens will be the 47 characters themselves. The reason is data. Every token's behaviour has to be learned from its appearances in the text, and our corpus is tiny — if we cut it into hundreds of subword tokens, most of them would appear only a handful of times, too rarely to learn anything from. With 47 characters over 83,000 characters of text, every token gets thousands of appearances.With 36 characters over 16,000 characters of text, every token gets hundreds of appearances. (Yes, this means generation is slower, as we said — but at our scale, the too-little-data problem outweighs the too-many-steps problem.) Character-level is the right choice at our scale; BPE is the right choice at the scale real models train at. (If you want to see how BPE actually finds these pieces, there is an aside at the end of this write-up.)

What does the model do?

Our machine has exactly one job: given a set of previous tokens, predict the probability of every token in our vocabulary appearing next. In other words: looking at the tokens so far, out of the 47 tokens we have, which one has the maximum probability to appear next?

The machine we are going to build is essentially going to give out exactly this. Once it generates one token, it becomes a recursive loop: you feed the whole thing back in again. "அகர முதலin these dark waters" goes in, the next token comes out, the extended text goes in again — and you keep going until you hit a limit you have set, or some stop condition. We keep generating tokens based on probability, and that is the system we are trying to build. That is what the model does. Every section below is about how it does it.

Tokenization

With the vocabulary built, we need a way to denote it that is comprehensible by machines. That cannot be characters; it has to be numbers. We are going to assign something called a token ID to each of our 47 characters. The IDs can simply start from 0 and go up to 46, and each character gets one. Henceforth, ப will not be "ப""s" will not be "s" — it will be denoted by its token ID.

You might ask: why do we need to convert them into numbers at all? Because we need to do a lot of calculations on top of these tokens, and computers calculate with numbers. A lot of arithmetic is coming, so numbers are the way to go.

A follow-up doubt: is the index number — say 16 — what we are going to do the arithmetic with? No, that is not correct. The ID is just an address. What we will actually calculate with is an embedding that we create for each token. So what is an embedding?

Embedding

We want the machine to understand the relationships between words — which ones are similar, which ones go together, and things like that. So for each token, it needs to store a profile from which such relationships can be read off — and that profile is called an embedding. (Note: the relationships themselves are never stored anywhere. Tokens that behave similarly simply end up with similar profiles, and that is the relationship — it emerges from comparing profiles, like inferring friendships by comparing personality questionnaires.) An embedding is a vector, and the size of the vector is our choice. We pick a size, and we initialize the vector to a random point. We do this for every token in the vocabulary — so what we really have is a table of 47 vectors, one row per token. Keep this table in mind; it is the first of the model's weights we have met.

When we say we "train" an LLM, these embeddings are among the things that get updated — and they are just numbers in a vector. Say the vector is of size 1024. You can imagine a multidimensional space of 1024 dimensions, and a single token placed as a point in that space. In a 3D space you need 3 values to place a point — x, y, and z. In this space you need 1024 values — and as the model learns from the text, each of those values comes to signify something. That is why we need an embedding.

Attention

By now, every token is a vector. But how do vectors turn into the probabilities we promised? Not randomly — everything that is so special about the LLM comes from the paper called "Attention Is All You Need."

Let us take an example (using English words as the tokens here, purely for readability): "The dog is going for a" — and we are supposed to predict the next token. Now, to predict it: which tokens in the past are going to be important, and what weightage will each of them carry? "dog" — the subject — is going to be very important, and maybe "going … for" tells us the action underway. These tokens are probably going to carry the most weightage in predicting the next token.

In our training, what we are going to do is make the system somehow learn the weightage of the previous tokens on the new token — that is what it is going to learn.

But how is this calculation actually done? It is built on three things called Q, K, and V: query, key, and value.

Let's say we are at "The dog is going for" — the latest token is "for", and it is for "for" that we now do the calculation. "for" is going to ask a query. The query can be anything; think of it as "who here is relevant to what I need?" Each of the tokens so far — "the", "dog", "is", "going", and "for" itself — has a key. (Yes, "for" included: a token is allowed to consult itself, and its own identity is often the biggest clue. Only tokens that come after it are out of reach.) The key is its identity — what it has to offer. And each token also carries a value — its actual answer, the thing it will contribute if chosen. The machine's job is to figure out which of these tokens has the best answer for this query.

For example, if the question is "who is the subject here?", the "dog" token should win. We know that because we've learned language — but how does a machine find out? It compares the query with each token's key by multiplying the two vectors — a dot product. A dot product in mathematics tells you how aligned two vectors are: the more aligned a token's key is with the query, the more relevant that token. Whoever aligns best carries the most weightage — and it is their value that gets passed along, in proportion to that weightage. That is how the relevance calculation happens.

One hard rule before we go further: the future is out of reach. "for" may consult every token up to and including itself — but never anything that comes after it. When the model is generating, that is automatic: the later tokens don't exist yet. During training, though, the full text is sitting right there — so the later tokens are deliberately hidden from each position, because a model that can peek at the answer never learns to predict it. This rule has a name: the causal mask.

Behind the query, key, and value sit three matrices — call them the query matrix, key matrix, and value matrix — initialized to random small numbers to begin with. How does the model know what the query should be, what each key should be, and what a value should return when a key matches a query? What is the question I should even ask? All of that is what the model learns when it goes through training.

As an example, take "The dog is going for a". The word "a" — the current token — has its embedding, and the query matrix gets multiplied with it to get the query. Every token's embedding — "a" included — gets multiplied with the key matrix to get its key, and with the value matrix to get its value. You multiply a matrix with a vector, and you get another vector. Now, how does all of this end up as a probability for each of the 47 tokens?

Now, getting into some interesting calculations. The dot product of the query vector and each key vector gives us a number, and the magnitude of that number tells the weightage of each past token on the current token. We want these to behave like percentages, so that we can say what share of influence each past token carries.

How do we go from raw magnitudes to percentages? There is something called softmax. You raise e to each of these numbers (this makes everything positive and stretches the gaps), then divide each by the sum of them all — and percentages come out. So it is like: 35% dog. Now what you do is multiply each percentage — 35% is 0.35 — into that token's value vector, do this for all the past tokens, and add them all up. You get a single vector.

That single vector now holds a description: "these are the attributes of the token I am expecting next." How do we go from that description to actual tokens? We already have a vector for every candidate — the embeddings of all 47 vocabulary tokens. So we take the dot product of our description vector with each of the 47 embeddings — the same alignment trick again — and each one gives a number: 47 scores (these raw scores are called logits). And again, from raw numbers to probabilities: softmax, one more time. We end up with 47 probabilities for the next token, and that is how the calculation is done.

Positional encoding — order matters

Now let's add a missing piece: positional encoding. We saw that with "The dog is going for a", the current token asks a lot of questions of the previous tokens. But there is one thing we ignored: the order in which the tokens come. "The man bit the dog" versus "the dog bit the man" — if you don't consider order, both hand the model the same bag of tokens, and the next token would get the same weightages. So the order has to be considered. The fix: just like the embedding vector for each token, we keep a positional vector for each seat — seat 0, seat 1, seat 2, and so on — and each token's input becomes its embedding plus its seat's vector. Now the same token in a different seat is a different vector, and order flows into all the calculations. These positional vectors also get updated during training — they are one more of the weights.

Training: where the learning happens

So now you ask: where does the learning happen? Remember, we calculated the probability of what the next token should be — and during training, we also know what the next token actually is, because it is right there in the text we are learning from. Based on how far off our probabilities were, the embedding table and the query, key, and value matrices get adjusted a little. (Note: the matrices get adjusted — not the per-sentence vectors; those are recalculated fresh every time.) That is what learning is. And everything we calculated happens again for each iteration of the learning: we do this again and again and again, and the machine somehow, magically, comes to understand how to produce language. That is the beauty of it.

But how does the model know whether it predicted a right value or a wrong one? That is where the concept of a loss function comes in. Let's say the model predicted something utterly crap instead of the correct token — then we get a huge value from the loss function, so the model knows it is off by a very big margin. Blame for that error is traced backward through all the calculations we did, to figure out which numbers in the embeddings and the query, key, and value matrices should move, and in which direction — that backward blame-tracing is called backpropagation. Adjust a little, try again, and keep doing this again and again so that the loss keeps reducing.

And now one might think: what if the model just memorizes the text? That is where the surprise test comes in. We don't show the model the entire text — we hide 10% of it (the hidden part is called the validation set), and we measure the correctness of predictions on the hidden text as well. If the loss keeps improving on both the training text and the hidden text, the model is genuinely learning. But if the training loss keeps reducing while the hidden text shows no improvement, we know the model is cheating us by memorizing. (This cheating has a name: overfitting.)

Sampling: why the output is different every time

Why does a language model not produce the same tokens again and again for the same prefix? If I say "The dog is going for a", it does not give the same continuation every time — it gives different things. The trick lies here: we get a probability for each of the 47 vocabulary tokens, but we do not always pick the top one. Sometimes we pick the second-most probable, sometimes the third — we roll dice according to the probabilities. That randomness is how a model gets creative. How adventurous the dice are is controlled by a value called temperature, and how many of the 47 even stay in the running is controlled by a value called top-k.

Model size and context length

Whenever you hear people say "an 8-billion model" — those 8 billion numbers are these weights: the embedding table, the positional vectors, the query, key, and value matrices, and all the other matrices. That is the model's size.

Context length is a different measure: the number of previous tokens the model can attend to in a single pass. This impacts the memory and the capacity needed to run the model — remember, every new token's query looks back at every previous token's key, so the longer the context, the more pairs to compute. (Note that these are two separate dials: an 8-billion-parameter model can come with a small or a huge context length — knowing more and seeing more are different things.)

Multi-head and multi-layer: the two multipliers

Everything described so far is a complete, working language model. What real models add on top are two multipliers — more heads, and more layers.

Multi-head attention

So far, from each token we calculated one query, one key, one value — a single set of QKV matrices. That one set tends to capture one kind of feature — probably the subject, what the text is talking about. But maybe we also need to understand the tone, the sentiment, the order, the way things are written. So people actually keep multiple such query-key-value matrices, each set initialized to its own different random small numbers — for the simple fact that there is so much to learn. Each set is called a head; each head runs the same calculation we described, in parallel, and their answers are combined. That is multi-head attention, and it helps the model learn many aspects of the past at once when predicting the next token.

Multi-layer

All the calculations we did so far make up one layer — and within it, the multi-heads ask their different questions in parallel: who is the subject, what is the sentiment, the order, and other things. Multi-layer is a different move: not asking more questions, but going deeper into what the answers mean. The output of one layer becomes the input of the next — and here is the point: layer 2 is not re-reading the raw tokens; it reads layer 1's conclusions. Going into layer 1, "dog" was just a token; coming out, it has become "dog — the subject — of an in-progress going". So layer 2 can ask questions that could not even be phrased before, questions about the relationships between relationships. Every layer is one more re-reading of the text, each starting from a deeper understanding than the last.

(Footnote: the embedding vectors plus positional encoding are used only as layer 1's input; from there on, each layer's output vectors are the next layer's input. And the final probability calculation over the 47 tokens happens only once, after the last layer — everything in between is enrichment.)

That's the whole machine

And that is genuinely all of it. A vocabulary. A table of profiles — the embeddings. Seats, so that order matters. Attention asking "who here matters right now?" and blending the answers. A loss that measures how wrong the guess was, and training that nudges every weight to make it a little less wrong. Dice that pick the next token. Feed the output back in, and language comes out. Everything else you will hear about LLMs — more heads, more layers, bigger tables, longer contexts — is this same machine, multiplied.

A caveat: this is not an assistant

One honest note before we close. What we built here is not a question-answer type of LLM. Ask it a question and it will not answer you — it will simply continue your text, because continuing text is the only game it knows. Everything in this write-up is what is called pre-training: teaching a model to predict the next token from raw text.

What you see as ChatGPT or Claude has gone through much more after this stage — post-training. The fluent next-token machine is further trained on curated conversations, so it learns that text shaped like "user asks → assistant answers helpfully" is what it should be continuing. Then it is tuned on human and AI feedback — techniques called RLHF and RLAIF — toward the kinds of answers people actually prefer. That is what turns a text-continuer into an assistant. The capability comes from pre-training; the helpfulness comes from post-training.

Aside: how BPE actually works

Still, it is worth seeing how BPE finds the middle path, because every model you will actually use runs on it. The way the algorithm works is that you look at every pair of adjacent characters in the corpus and count how frequently each pair appears. The most frequent pair gets fused into a new token. For example, in Tamil, ன (na) + ் (the dot on top) makes ன் — and that pair appears so often that it becomes a new token. (In our corpus this pair was merge #3, appearing 1,272 times — the very first merge was even humbler: ் + space, seen 3,552 times.)For example, in our haiku corpus, "." + "." makes ".." — the book's beloved " ... " pauses are so frequent that this was literally the first merge: the pair appeared 392 times. Letter pairs like "th" and "in" follow right behind. Then you repeat the process — and since the fused tokens can now pair up with others, the pieces keep growing.

You fix the vocabulary size up front — a few hundred for a tiny corpus like ours, or 100,000+ for a frontier model — and run this loop until the vocabulary is full. That gives you tokens that live in between characters and words. In our build, though, our tokens remained the 47 characters.