Notes on working with AI
Nobody wrote the code that makes Claude write Python
Calling a language model autocomplete is accurate about its objective and wrong about its capability. The pipeline explains why, and what it means for how you use it.
A language model is trained on one task, and nothing else is programmed
No one wrote the code that lets Claude produce working Python or translate French. There is no module for either. What exists is a single objective, predicting the next token, applied to a very large body of text and code. Every ability the model has came out of optimizing that one number.
The pipeline starts with data. Text is collected, deduplicated, filtered for quality and cleaned. Then comes tokenization. The model does not read words. It reads tokens, which may be a whole word, a fragment or a punctuation mark, and each token is mapped to a vector of numbers called an embedding.
The transformer decides which context matters
The network is a transformer, a stack of layers whose key mechanism is attention. Attention lets each token weigh every other token in the context and pull in the ones that matter. In the sentence "the programmer fixed the server because it crashed", attention is what ties "it" to "server". No grammar rule does that work.
Pre-training turns prediction error into capability
Training begins with random weights. The model predicts the next token, a loss function measures how wrong it was, backpropagation assigns that error to individual weights, and gradient descent nudges them in a better direction. Repeat this across a vast dataset on thousands of GPUs for weeks. This is pre-training, and it is where most of the model's knowledge and ability originates.
What comes out is a very strong autocomplete. Ask it a question and it may continue with more questions, because that is a likely continuation of text.
Post-training shapes behavior on top of that foundation
Three techniques turn the text continuer into an assistant. Supervised fine-tuning uses human-written pairs of question and ideal answer. Reinforcement learning from human feedback compares outputs and rewards the better ones. For reasoning models there is also reinforcement learning with verifiable rewards: the math answer matched or it did not, the code passed its tests or it did not. That last signal matters because it does not depend on anyone's judgment.
Serving is a separate engineering problem. The model is sharded across many GPUs, its weights are quantized to use less memory, and requests are batched so thousands of users share the same hardware. That makes the model fast and affordable. It does not change what the model knows.
Two practical consequences
Output is conditioned on context. The model produces the most plausible continuation of what sits in its window, so the window is the lever. Material you supply raises the quality of the guess. Material you leave out gets filled from general patterns, which is why a bare request returns a generic answer.
Fluency is not a correctness signal. Fluent text is exactly what the objective rewards, so a wrong figure and a right one arrive in the same confident tone. Anything that carries a number, a citation or a legal claim needs a check that lives outside the model. Treat the first answer as a draft.
Why "just autocomplete" misleads
The phrase describes the objective accurately. It says nothing about the capability, because capability was never specified anywhere. Writing code, explaining physics and translating between languages emerged from one simple objective repeated at enormous scale: predict what comes next, and be slightly less wrong each time.
The objective also tells you nothing about the ceiling. I would be wary of anyone who claims to know exactly where it is.
What to do tomorrow
Ask Claude about itself:
Briefly explain how a language model is built, then give me three rules for phrasing requests to you. For each rule, say when it will not help.
Read the three rules. Most of them will come down to giving the model more to work from.