How Large Language Models Work: Tokens, Attention and Sampling

A large language model (LLM) predicts the next piece of text, and everything it does, from answering questions to writing code, is that one operation repeated. This guide explains how it works, step by step, with three worked examples (tokens and cost, attention, sampling) that you can check by hand with a pencil. The examples follow Saathi, a customer assistant for a fictional bank that we use in our Generative AI course. The bank and its people are fictional, and this guide teaches engineering practice, not legal, regulatory or investment advice.

1. What an LLM is, and what it is not

A large language model (LLM) is a very large mathematical function with billions of adjustable numbers, called weights or parameters. Given a sequence of text, it outputs a probability for every possible next piece of text. That is the whole interface. Answering, summarising, translating and writing code are that one operation repeated.

The common question, "is it just clever autocomplete?", deserves a straight answer. The mechanism is next-token prediction, and the word that misleads is "just". To predict well across a vast range of documents, the network has to build internal representations of grammar, facts, styles and patterns of reasoning. They are real and useful. They are also compressed and approximate, like a student's memory of a textbook read once.

So here is what an LLM is not:

• Not a database. It holds no records you can query. A fact is smeared across millions of weights and may be recalled, blurred or lost.
• Not a search engine. Unless a tool fetches text for it, nothing is retrieved at answer time.
• Not a calculator. It imitates arithmetic from patterns; it does not execute it.
• Not a witness. It cannot say where it learned something, and it has no sense of "I never saw this". It produces the likeliest continuation either way.

What it is very good at is language-shaped work: rewriting, summarising a policy in plain Kannada, classifying a WhatsApp message, extracting fields, drafting a reply. A good Saathi design gives the model that work and keeps facts, arithmetic and actions in systems that can be checked.

2. Tokens: the model's alphabet

A tokenizer converts text into tokens, each an integer ID from a fixed vocabulary of tens of thousands to a few hundred thousand entries. Common words are often one token; rare words, names, numbers and non-English text break into several.

Most modern tokenizers use a form of byte-pair encoding (BPE). Start with single characters, find the most frequent adjacent pair in a large corpus, merge it into a new symbol, and repeat until the vocabulary is the size you want. The tokenizer replays the stored merges, in order, on new text.

Why it matters: providers price and limit text in tokens, not words. Context windows, output limits and invoices are all token counts. A language or script the tokenizer saw less of gets split into more pieces, so the same message costs more and fills more of the window. Petrov et al. (2023) showed that the same content can need several times more tokens in some languages than in others.

Worked example 1: counting tokens and costing a month

Suppose a toy tokenizer has this ordered merge table:

Take the text "loans emi" and tokenize each word separately, starting from single characters and applying merges in order.

• "loans" starts as l, o, a, n, s. Merge 1 gives lo, a, n, s. Merge 2 gives loa, n, s. Merge 3 gives loan, s. No other merge applies. 2 tokens.
• "emi" starts as e, m, i. Merge 4 gives em, i. Merge 5 gives emi. 1 token.

Total: 3 tokens for 8 letters. A word the table has never seen would stay as single characters.

Now the cost. Invented prices, for arithmetic only: ₹25 per million input tokens and ₹100 per million output tokens. A Saathi conversation uses 1,500 input tokens (instructions, retrieved text, history, the question) and 300 output tokens. For 40,000 conversations in a month:

• Input: 40,000 × 1,500 = 60,000,000 tokens, which is 60 million × ₹25 per million = ₹1,500.
• Output: 40,000 × 300 = 12,000,000 tokens, which is 12 million × ₹100 per million = ₹1,200.
• Total: ₹2,700.

Most of the input tokens are repeated instructions and context, not the member's question. That is why prompt length, not message length, drives cost.

3. Embeddings and cosine similarity

Each token ID is looked up in a table that returns a vector, a list of numbers (perhaps a few thousand). This is the token's embedding. Training adjusts the vectors so that tokens used alike point alike. As text passes through the network, each position's vector is refined to represent that token in context: "bank" near "river" and "bank" near "loan" end up different.

Separate embedding models give one vector per passage, powering semantic search. To compare two vectors you use cosine similarity:

cos(a, b) = (a · b) / (|a| × |b|)

where a · b is the dot product (multiply matching entries, add them up) and |a| is the length of a (square root of the sum of squares). The result runs from 1 (same direction) through 0 (unrelated) to -1 (opposite).

A hand check with two dimensions. Let the query "block my card" be a = (3, 4). A passage on lost cards is b = (4, 3), and a passage on loan interest is c = (4, -3).

• a · b = 12 + 12 = 24. |a| = 5, |b| = 5. cos = 24 / 25 = 0.96. Very close.
• a · c = 12 - 12 = 0. cos = 0. Unrelated.

Cosine measures direction, not length or word overlap. Two passages can share no words and still score high if they mean the same thing, which is why "mera card kho gaya" can find "report a lost card".

4. Attention: how distant words talk to each other

Earlier models read text in a chain and struggled to keep early words alive across long sentences. Vaswani et al. (2017), Attention Is All You Need, introduced the transformer, in which every position can look directly at every other.

For each token the network computes three vectors from its embedding:

• a query: what this token is looking for,
• a key: what this token offers to be found by,
• a value: the information it passes on if it is found.

The attention output for one token is computed in four steps:

1. Take the dot product of its query with every key. A larger dot product means a better match.
2. Divide by the square root of the vector size d. This scaling stops the scores from growing with dimension and pushing softmax into extremes.
3. Apply softmax to turn the scores into positive weights that sum to 1. Softmax of scores s is e^s divided by the sum of all e^s.
4. Output the weighted sum of the value vectors.

In symbols, softmax(Q K^T / sqrt(d)) V. A model runs many such attention "heads" in parallel, each free to learn a different relationship (what "it" refers to, which noun an adjective modifies).

Worked example 2: one attention step

Let d = 4, so sqrt(d) = 2. A query is q = (2, 0, 2, 0). There are three earlier tokens with keys and values:

Step 1, dot products: q · k1 = 2 + 2 = 4. q · k2 = 0. q · k3 = 2.

Step 2, scale by 2: scores are 2, 0, 1.

Step 3, softmax: e^2 = 7.389, e^0 = 1, e^1 = 2.718. Their sum is about 11.107. Weights are about 0.665, 0.090, 0.245. Check: they add to 1.

Step 4, weighted sum of values: first component 0.665 × 1 + 0.090 × 0 + 0.245 × 1 = 0.910. Second component 0.665 × 0 + 0.090 × 1 + 0.245 × 1 = 0.335. The output is about (0.910, 0.335).

Token 1 matched the query best, so its value dominates. This happens for every token, at every layer, in every head. The price is that every token is compared with every other, so pairs grow with the square of the length (Section 8).

5. The transformer block, conceptually

Think of a transformer as a stack of similar blocks, perhaps dozens of them. Each block does two jobs:

• Attention mixes information across positions: each token gathers what it needs from the others.
• A feed-forward network processes each position independently, transforming what attention collected.

Residual connections (each block adds its result to its input) and normalisation keep deep networks trainable. Because attention alone ignores order, position information is added to the embeddings. A causal mask ensures each token attends only to earlier tokens, not to the future it is about to write.

In order, the data flows like this: embeddings plus position, then block 1 (attention and feed-forward), then blocks 2 to N, then the final vector at the last position, then logits for every token.

Only the last position's final vector is used to choose the next token. Everything else exists to make that vector a good summary of what should come next.

6. Next-token prediction and sampling

The last vector is multiplied by a table the size of the vocabulary, producing one raw score, a logit, for every token. Softmax converts logits into probabilities. The model has now produced a distribution, not an answer. A separate decision, sampling, chooses one token. That token is appended to the input and the model runs again: autoregressive generation. Every token is a fresh draw conditioned on all earlier ones, including the model's own words, which is why an early mistake can steer the rest of a reply.

1. 1TokensThe text is cut into tokens
2. 2EmbeddingsEach token becomes a vector
3. 3AttentionTokens pass context to each other, layer by layer
4. 4LogitsOne raw score for every token in the vocabulary
5. 5SoftmaxThe scores become probabilities
6. 6SampleOne token is chosen, added to the text, and the model runs again

The sampling controls are:

• Greedy decoding always takes the highest-probability token: repeatable, but dull and prone to loops.
• Temperature (T) divides the logits by T before softmax. T below 1 makes the distribution sharper; T above 1 flattens it; as T approaches 0 sampling becomes greedy.
• Top-k keeps the k highest-probability tokens, renormalises, and samples from them.
• Top-p (nucleus) sorts tokens by probability and keeps the smallest set whose cumulative probability reaches p, renormalises, and samples. Unlike top-k, the set size adapts: small when the model is sure, larger when it is not.

Entropy, H = - sum of p × log2(p), measures how spread out a distribution is. Four equally likely tokens have entropy 2 bits; one certain token has 0. Lower temperature lowers entropy.

Worked example 3: temperature and top-p

Saathi has written "KCC loan ka byaaj" and the model scores four candidates for the next token:

At T = 1. e^2.0 = 7.389; e^1.0 = 2.718; e^0.5 = 1.649; e^-1.0 = 0.368. The sum is about 12.124. Probabilities: kam 0.610, zyada 0.224, kitna 0.136, nahi 0.030.

At T = 0.5. Divide each logit by 0.5 to get 4, 2, 1, -2. Then e^4 = 54.598; e^2 = 7.389; e^1 = 2.718; e^-2 = 0.135. The sum is about 64.840. Probabilities: kam 0.842, zyada 0.114, kitna 0.042, nahi 0.002. The leader grew from 0.61 to 0.84.

• kam61%
• zyada22.4%
• kitna13.6%
• nahi3%

• kam84.2%
• zyada11.4%
• kitna4.2%
• nahi0.2%

Top-p = 0.9 at T = 1. Cumulative probabilities in order: 0.610, 0.834, 0.970. The smallest set reaching 0.9 is the first three tokens. Renormalise by dividing by 0.970: kam 0.629, zyada 0.231, kitna 0.140. "nahi" can no longer be chosen.

Top-p = 0.9 at T = 0.5. Cumulative: 0.842, 0.956. Now the nucleus has only two tokens.

Top-k = 2 at T = 1. Keep kam and zyada, renormalise over 0.834: 0.731 and 0.269.

Two lessons. Temperature and top-p interact: sharpening the distribution shrinks the nucleus. And none of these controls changes what the model knows, only how boldly it draws from what it finds likely. Low temperature makes Saathi consistent. It does not make Saathi correct.

7. How the model was trained, and what each stage changes

In RLHF (reinforcement learning from human feedback, as in Ouyang et al., 2022, the InstructGPT work), people rank candidate answers, a reward model learns those preferences, and the language model is then optimised to score highly. DPO (Rafailov et al., 2023) reaches a similar goal directly from pairs of preferred and rejected answers, without a separate reward model. Constitutional AI (Bai et al., 2022) uses written principles and AI feedback to scale part of the process.

Two consequences matter. Preference tuning rewards answers that look good to raters, which can include sounding confident, so a fluent wrong answer is a natural by-product. And later stages change behaviour, not knowledge: a fact absent from pre-training is not added by instruction tuning. LoRA (Hu et al., 2021) makes small adapter updates cheap, which makes fine-tuning for a house style practical, but it is a poor way to keep a changing rate card current.

8. The context window and the cost of length

The context window is the maximum number of tokens (your input plus the output) the model can consider in one request. Inside it, everything is tokens: instructions, retrieved documents, history, the member's message.

Length has three costs:

1. Money. You pay per token, on every call, for everything you send again.
2. Compute and delay. Attention compares every token with every other, so pairs grow with the square of the length. A 2,000-token prompt has about 4 million pairs; an 8,000-token prompt about 64 million. Four times the length is sixteen times the pairs. Caching earlier tokens' keys and values (the KV cache) softens this but never removes it.
3. Quality. Liu et al. (2023), Lost in the Middle, found that models tend to use information at the start and end of a long input better than in the middle. A bigger window is not better memory. Put the critical rule at the edges, keep the context short, and test where facts sit.

This is why retrieval exists: send the few relevant passages, not the whole archive.

9. Scaling, reasoning and multimodal models

Scaling. Next-token prediction loss improves smoothly and predictably as model size, training data and compute grow. Hoffmann et al. (2022) showed that, for a fixed compute budget, model size and the number of training tokens should grow together, and that many large models had been trained on too little data for their size. So bigger is not the only lever. Scale improves many skills but has not removed hallucination, because the objective is still plausible text.

Reasoning-style models. Some models are trained to produce a long chain of intermediate steps before the visible answer, spending extra computation at answer time, called test-time compute. This helps on multi-step problems, such as multi-condition eligibility. The costs are higher latency, more billed tokens, and no guarantee of truth, since a long chain can start from an invented premise. Route hard cases to them; do not make them the default.

Multimodal models. A multimodal model accepts more than text, such as images or audio. These are converted into embeddings that sit in the same context as text tokens, so the same attention machinery can relate a photo of a cheque or a Kannada voice note to a written question. The same cautions apply: a blurred amount is read as a plausible guess, so critical fields still need checks.

10. Why hallucination is a property of the method

Imagine asking a model for a regulator's circular on a niche topic. In training data, circular references have a recognisable form: a department code, a year, a serial number, a subject line, a few numbered rules. The model learned that form very well and can produce a perfect-looking circular. Whether that particular circular exists never entered the calculation. Plausible and true are different properties, and the training objective rewards only the first.

Four things follow.

• Fluency is not evidence. A confident, well-formatted answer is what the system is optimised to produce, right or wrong.
• Asking again does not verify. The model's "yes, that is real" is more generated text. In Mata v. Avianca the chatbot assured the lawyers the cases could be found in legal databases; they did not exist.
• Risk is highest in the long tail: specific numbers, names, citations, recent events and bank-specific facts that were rare or absent in training.
• It can be reduced, not eliminated. Supply the source text and say to answer only from it; require citations that code can check; allow "I don't know" and route to a person; validate numbers against systems of record; test with questions whose answers are not in the context.

Knowledge in weights versus knowledge in context

Knowledge in the weights was learned in training. It is frozen at a cutoff, cannot be quoted, and contains nothing bank-specific. Knowledge in the context is text you supply with the request. It is as fresh as you make it, it can be quoted, and a missing or ignored passage is a failure you can detect, unlike a confident invention.

For Saathi: use the weights for language skill (understanding Kanglish, writing polite replies) and the context for every fact that could appear in a complaint or an audit.

11. Known limits to design around

• Knowledge cutoff. The weights stop absorbing new information at a point in time. Anything later, including a rate change last month, is invisible unless you supply it.
• Arithmetic. Digits are tokens, and multi-digit arithmetic is pattern imitation. Simple sums often come out right; long or unusual ones do not. Check: ₹1,25,000 at 7.5% simple interest for one year is ₹9,375, yet a model may write ₹9,357 with the same confidence. Give calculations to code or a tool.
• Long-context degradation. Quality can fall as input grows, and middle facts are used less reliably.
• Sensitivity and non-determinism. Small changes in phrasing can change answers, and even at low temperature outputs may differ between runs and model versions. This is a reason for tests.
• Language gaps. Performance is usually strongest in the best-represented languages; Kannada and code-mixed Roman-script text need their own testing.

12. What to carry forward

A sentence you could say to the board: "Saathi is a language engine that predicts likely text. It is strong at understanding and drafting in Kannada, Hindi and English, and unreliable as a source of facts. So the facts it states will come from bank text we supply, and what it cannot ground, it must say it does not know."

A checklist to keep:

1. Is the fact in the weights or the context? If it matters, put it in the context.
2. Can a person or a program verify the output against a source?
3. How many tokens does this design send on every call, and who pays?
4. What happens when the model does not know? Is there a polite, useful "I do not know" path?

The next step is practice: how to speak to a model so that it behaves the same way every time, how to ground it in your documents (see RAG explained), and how to measure it. That is the path of our Generative AI & Agentic AI course.

Sources and further reading

• Vaswani et al. (2017), Attention Is All You Need.
• Petrov et al. (2023), Language Model Tokenizers Introduce Unfairness Between Languages.
• Ouyang et al. (2022), Training language models to follow instructions with human feedback.
• Rafailov et al. (2023), Direct Preference Optimization.
• Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback.
• Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models.
• Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts.
• Hoffmann et al. (2022), Training Compute-Optimal Large Language Models.