A language model understands nothing. There is nothing inside it that resembles the meaning of a word, it does not "know" anything, it does not "want" anything. And yet it works.
This article starts there, because the most useful thing to understand about this technology is why a machine that understands nothing manages to produce text that seems to understand. The answer is simpler and deeper than the industry is willing to admit, and it can be explained in a paragraph.
Then there are models, agents, and the difference between them, which is the part that is talked about least and the one that matters most when you decide whether to use something in a real project.
What is inside: only numbers
The basic mechanism is the embedding vector. A word is represented as a list of floating-point numbers, about ten for common words, hundreds for the more refined ones.
The useful analogy is not "meaning", it is position. Imagine a map of a city where every point is a word and the distance between two points represents how similar they are. On this map, "dog" and "cat" are close, and "dog" and "biscuit" are less close, even though neither of the two has anything to do with the other in terms of meaning.
The fundamental point is this: the similarity between words is a statistical consequence of how they are used, not a fact about the world. The model does not know what a dog is. It knows that in billions of sentences the word "dog" appears near "to bark" more than "calculator".
And there is a consequence that must be understood well because it explains almost all the failures we talk about: there is no such thing as "the meaning of a word" in a model. There is its statistical distribution in a vector. They are different things.
How it works: next-token prediction
The heart of the system is incredibly simple, and its simplicity is the good news.
The model has one single task: given the text so far, what is the most probable token that comes next? Everything else is technique for answering this question well.
" Il gatto dorme sul " -> probabile: tappeto (35%)
possibile: divano (18%)
possibile: pavimento (9%)
possibile: letto (6%)
Il modello sceglie, lo aggiunge, e ripete da capo.
It is called autoregression, and hence the name "generative model". There is no module that "understands the text" and another that "writes": there is a mechanism that guesses the next piece, applied a thousand times.
From this three properties follow that explain everything:
why it makes mistakes in a convincing way: an error is not a random error, it is a wrong but plausible prediction;
why it does not know when to stop: the end of the text is a token like any other, predicted with a probability;
why it cannot be asked why: there is no reasoning that produces the answer, there is a probability distribution.
How it is structured: the transformer
If the task is to guess the next token, the hard problem is understanding which word it refers to. In "The dog ate the biscuit because it was hungry", the word "it" depends on much further back.
The 2017 solution is called attention, and it is this: every token, at every step, asks all the others "how important are you to me?" and weighs the answers.
" Il gatto ha mangiato il biscotto perche era affamato "
"perche" -> si appoggia a "mangiato", "gatto", "biscotto"
"era" -> si appoggia a "gatto"
"." -> si apposta su tutto il fraseggio
Ogni parola guarda TUTTE le altre e decide a chi dare peso.
E' il punto in cui il contesto entra nel calcolo.
Each block of the transformer does three things: multi-head attention (several types of relation in parallel), a normalisation step, and a network that transforms each token. Dozens or hundreds of blocks are stacked, and the information is refined layer after layer.
A non-obvious detail that is worth it: attention on its own does not distinguish the order of words. This is why positional encodings are needed, which inject position information. If you removed them, "dog bites man" and "man bites dog" would become the same thing.
And training is, in its turn, a prediction. You show the model a text, you ask it for the next token, you calculate the error, and you correct the weights. Repeated over trillions of tokens. Training is the prediction task, carried out on a gigantic scale.
The difference between model and agent
It is the distinction that is almost always missing, and it is the one that decides whether something is useful in a project.
| Model | Agent |
|---|---|
| What it is | a function: text in, text out |
| Input | a prompt |
| Output | a response |
| Feedback | none |
| Persistence | none |
| It fails | by producing a wrong response |
| Example | "translate this" |
The point that must be said clearly: an agent is not a more intelligent model. It is a model that repeats a loop, and the loop is what adds the risk.
The loop is simple to describe and you need to know it by heart:
# pseudocodice di un agente
mento = obiettivo_iniziale
while non ho finito and not tentativi < 10:
azione = modello(mento, obiettivo) # "provo a fare X"
risultato = esegui(azione) # un tool vero gira davvero
mento += "X ha prodotto: " + risultato # il mondo risponde
risposta = modello(mento) # "ho finito, ecco cosa ho trovato"
The difference is the line risultato = esegui(azione). A model has no
effects on the world. An agent does: it calls an API, writes a file,
deletes something. And a wrong action is not a wrong response to be
corrected: it is damage.
From this follows the most important practical consequence of the whole article:
The more autonomy you give an agent, the more prolific it becomes. And it is in exactly the same proportion that the surface for damage grows.
An agent that can only read is a privacy problem. One that can write to a database is an integrity problem. One that can execute commands is a problem that concerns production.
The difference between models, which is a difference in form
Model families are not all the same, and the useful distinction is not the brand.
Text-only models accept text and return text. It is the simplest form and the one on which everything else is built.
Multimodal models also accept images or audio. It is not a convenience detail: a model that sees an image can be wrong about things you cannot check with the naked eye, and that is why human verification is worth more, not less.
"Small" and "large" models are the same architecture at a different scale, with a practical limit that no one states often enough: small models run locally, on your own machine, without a network. On confidential data, this is the difference between being able to use a model and not being able to use it at all.
Specialised and general models. A model fine-tuned on a small, well defined domain often beats a larger general model, because the advantage is not in knowledge but in the precision of the expected form. In a security context this is worth more than elsewhere: a model that knows what a CVE is is useful, a general model that half knows it is noise.
Models with tools. A model that can call a function is different from one that cannot, and this is the distinction between model and agent that I made above.
Why it works and why it fails
The two sides of the same coin, and they should be read together.
Why it works. Language has a statistical structure at many levels, and that structure contains real information about the world. Not the whole world, but enough to make translation, summarisation, classification and code possible. The model does not understand the meaning of the sentence: it has learned that certain sentences follow one another, and in certain contexts statistical correctness coincides with the correctness of meaning.
Why it fails. The same mechanism that produces correct answers produces false answers with the identical fluency, because the form is right in both cases. Hence the four properties you must keep in mind when you use one of these systems:
It does not distinguish true from plausible. A false answer has the same probability of seeming right as a true one.
It has no memory of its sources, so it cannot tell you where it got something. A citation without a reference is invented.
It does not know what it does not know. The confidence is uniform: there is no way to ask it "are you sure?", because there is no internal process that assesses certainty.
The context window is a real limit, and the tool talks about it little. When something "it does not see", often it does not see it because it is outside the window, not because it does not exist.
And there is the point concerning security, which must be said without jargon: you do not put secrets into the prompt of an external service. What you write has been sent to someone, and the promise that it is not saved is a commercial promise, not a technical guarantee. The local model does not have this problem, and it is one of the main reasons why it exists.
A bit of history
1958, the perceptron. The first artificial neural network, by Frank Rosenblatt, which learned to classify with the same principle as today's deep networks, but with a single layer. The definition is the classic one: a classifier that decides whether an input, represented as a vector of numbers, belongs to a class or not. It remained a laboratory subject for years, because there was no way to train it on large enough data.
1964-1967, ELIZA. Joseph Weizenbaum at MIT writes the first chatbot. It understood nothing: it associated patterns and substituted words, and the most famous script, DOCTOR, simulated a Rogerian psychotherapist. Weizenbaum was scandalised when he realised that users attributed understanding to it, to the point of realising that he had understood something about the human relationship that he had not understood about the machine.
It is the most important lesson in all of history, and it still holds: the first crude mistake about this technology was not that machines did not understand, it was that humans saw understanding in them. Every so often that book should be reread.
2017, the transformer. Google Brain publishes "Attention Is All You Need", by Ashish Vaswani and colleagues. The paper proposes the architecture that today underpins almost everything: it removed the sequential dependence that made training very slow, enabling the parallelism that then made enormous models possible. In the same paper the complete specification of the attention mechanism is already there.
This is the real turning point, and it must be said: the leap did not come from a mysterious idea, but from an ingenious idea that made parallelism economical.
2020, GPT-3. OpenAI releases GPT-3, a model with 175 billion parameters. The demonstration that matters is just one: with 175 billion parameters and no specific training, the model can do things it had not been programmed for, and in few shots. It is the first time the improvement came from scale and not from architecture.
30 November 2022, ChatGPT. The release that changed the public. The only thing it added compared to GPT-3 was the conversational interface, and its public availability. The model was not new, the way of using it was.
Today. Research has moved to scale, efficiency and multimodal capabilities, and the term "agent" has entered everyday language to indicate a system that uses a model and acts on the world. It is the operational definition I gave at the beginning, and the one that matters.
The rule we use
CYBER ONIONS' reasoning on all of this history is a single sentence:
A model produces text. An agent performs actions. The first can be corrected by hand, the second causes damage that cannot be undone.
Three operational consequences, which apply to any project in which these things enter:
start from the model, not from the agent. Add autonomy only when what you already have is measurable and finite;
the agent's tools are its security perimeter, not its instructions. The prompt tells the model what to do; the permission you give it tells it what it can do. The second matters more than the first;
always verify with tools, not by eye. A result must be checked with a command that returns a datum. It is the same rule as in the Vademecum dell'hacker: no verification done by eye is a verification.
The underlying reason we already know: the layered model teaches that every layer excludes those below it. A language model excludes nothing: it gives you an answer that seems complete without your knowing which part it made up. It is the same structure, with one risk more.
If only one thing remains from all this: a model is a writing tool, not a source. Everything else is a matter of how much you trust it before verifying.