Conceptors: An Introduction from First Principles

·Before we start

Who this is for, and a promise

This article is written for a curious, educated reader who has not done anything technical in a long time. You may remember, at least vaguely, what a t-test is and what a line of best fit is; that is all the background it assumes. You have probably never heard the words "encoding," "layer," "steering," or "conceptor" used the way this article uses them.

Here is the promise. If you read this in order and do not skip the early sections (even where they feel slow), then by the end you will be able to explain, in your own words and with a picture in your head:

1. what it means to encode a word or a sentence as numbers;
2. how a large language model is trained, and why that is really just the line of best fit with more knobs;
3. what the layers of such a model are, and why the middle ones matter most for our purposes;
4. what steering a model means;
5. what a conceptor is;
6. what steering can do that no amount of careful prompt-writing can do, and what a conceptor adds to steering.

Each step leans on the one before it. Losing the thread early makes everything later feel like magic, so the early sections are deliberately thorough, and important ideas are restated more than once, from different angles. In the later, harder sections you will see boxes labeled Another way to think about it; those are second doors into the same room, in case the first one was locked for you.

What you need to know already

Everything in this article reduces to a handful of things most people met in school or college and still half-remember:

• Averages, and differences between averages. The single most important object in this article, the "steering arrow," is literally the difference between two group averages: the top half of a t-test.
• Spread. The idea that a set of numbers has a typical distance from its average (a standard deviation). A conceptor is built from how much a set of examples spreads out, direction by direction.
• The line of best fit. Training a language model is the same idea as fitting a line: adjust the knobs until the error is as small as you can get it.
• Tables of numbers. Rows and columns, like a spreadsheet. Whenever this article says "row of numbers," picture one row of a spreadsheet; whenever it says "table," picture the whole sheet. Nothing more exotic is needed. The article shows the one formula of the research literature exactly twice, in Section 5.2 and in the glossary, for readers who want to connect what they read here to the papers; it is never needed to follow the argument.

How the illustrations were made

Most explanations of this material use cartoon diagrams: a box labeled "Encoder" with an arrow to a box labeled "Decoder." Those pictures do not help anyone who does not already know what is in the boxes. This article avoids them. Instead, nearly every figure was produced by running a real language model and plotting what came out.

The model is GPT-2 (the 124-million-parameter version released by OpenAI in 2019). It is small and old by today's standards, which is a feature here: it is small enough to run on an ordinary computer, its internals are fully public, and yet it displays every phenomenon this article needs to show. Where this article quotes results on bigger, newer models, it says so and cites the source. A short appendix describes exactly how each experiment was done.

Contents

1Words as numbers (what "encoding" means)

A computer cannot do arithmetic on the word coffee. It can only do arithmetic on numbers. So the first thing a language model must do is turn every word into numbers, in a way that keeps the word's meaning. That is what encoding is. This section builds the idea slowly, because everything later stands on it.

1.1 You already encode things

Think of a credit score. A person's entire borrowing history is squeezed into one number, 300 to 850, arranged so that nearby numbers mean similar risk. Two applicants with scores of 740 and 750 will be treated much the same; 740 and 550 will not. The number is not the person, but it carries the part of the person that a lender cares about. That is an encoding: a fixed rule for writing something as numbers so that the numbers preserve what matters.

One number is rarely enough. A house listing uses several: square footage, number of bedrooms, year built, distance to the nearest school. Together those four numbers are a small row in a spreadsheet, and two houses with similar rows are similar houses. Wine reviewers do the same with sweetness, acidity, tannin and body. A personality test turns a person into five numbers. A doctor's chart turns a patient into a row of vital signs. In every case, the trick is the same:

Figure 1 shows the smallest possible version with foods. Each food gets two numbers, a sweetness score and a saltiness score (rough, made-up values, just to make the point). With two numbers per food, each food becomes a dot on a page. Notice three things about the picture, because all three will come back for language models:

1. Similar foods land near each other. Pretzels sit next to potato chips; honey sits next to cake.
2. Distance is a fair measure of similarity. Bread is closer to plain rice than to honey, and that matches intuition.
3. Directions on the page mean something. Moving to the right means "sweeter." Moving up means "saltier." The orange arrow is not a food; it is a direction, and it has a meaning of its own.

1.2 Nobody chooses the columns

For foods, a person chose the two columns, sweetness and saltiness. For a language model there is no such person. The model uses hundreds of columns per word (768 in the model used for this article, and several thousand in today's large models), and no human decided what any column means. The columns are filled in automatically during training, by a process described in Section 2. For now, take it on faith that there is a way to fill them in so that words used in similar ways get similar rows.

The principle behind it is old and has a slogan from the 1950s: "You shall know a word by the company it keeps." The words coffee and tea appear in the same kinds of places: after "a cup of," before "with milk," near "hot," "drink," "morning." The words coffee and truck do not. A procedure that gives similar rows to words that keep similar company will, without anyone telling it so, end up putting coffee next to tea and far from truck.

1.3 A word really is a row of 768 numbers

Figure 2 shows real rows from the real model. GPT-2 keeps a lookup table with one row per word-piece it knows (50,257 rows) and 768 columns. The figure shows the first 16 of the 768 numbers for seven words. This is exactly what the model "sees" when it reads the word cat: not the letters c-a-t, but the row of 768 numbers that begins 0.01, 0.04, 0.16, −0.22, 0.03, −0.10, −0.34, −0.08, …

Two things to notice. First, no individual column has an obvious meaning. Column #7 is not "furriness." The information is spread across the whole row, the way a person's creditworthiness is spread across many line items rather than sitting in one. Second, the rows look like noise to a human eye, and that is fine: the model never looks at one column in isolation either. It only ever combines all 768 at once, and at that level the resemblance between cat, dog and kitten is strong and measurable, as the next subsection shows.

1.4 Similarity is closeness, and we can measure it

With two numbers per item you can see closeness on a page. With 768 you cannot draw the page, but you can still compute closeness. The standard score is computed almost exactly like a correlation coefficient between two rows: it comes out near 1 when the two rows rise and fall together, and near 0 when they are unrelated. (Technically it is the correlation without first subtracting each row's mean, and it is called the cosine similarity; you can treat it as "a correlation between two rows of numbers.") Here are real values from the model's lookup table:

The absolute values are not what matters (unrelated words sit around 0.2–0.3 in this model rather than at 0, because all words share some common "this is an English word" background). What matters is the ordering, which is consistently sensible. Figure 3 shows the same idea as a grid, and Figure 4 shows a map.

Pause on Figure 4 for a moment. The model was never given a dictionary, a list of cities, or a list of fruits. It saw only text. The clustering is entirely a by-product of "words that keep similar company get similar rows." That is what it means to say the numbers carry meaning.

1.5 Directions have meaning too

In the food picture, "to the right" meant "sweeter." The same thing is true in the 768-number space, and it is the single most important fact in this article, because steering and conceptors are both built on it.

Here is how to find a meaningful direction. Take a few words that mean "cold" (cold, freezing, icy, chilly, frozen) and average their rows. Take a few that mean "hot" (hot, warm, tropical, sunny) and average those. Subtract: hot-average minus cold-average. The result is a row of 768 numbers that is not a word at all; it is an arrow pointing from "the cold region" to "the hot region." Now take any other word, say Iceland, and ask how far along that arrow it sits. Do that for twenty places and you get a ruler (Figure 5, top).

Three rulers, three directions, none of them designed by a person. Siberia, Antarctica and Iceland at one end; Brazil, Australia and Spain at the other. Bread, noodles and water at one end; Mercedes, champagne, diamond and mansion at the other. Divorce, debt and defeat at one end; gift, victory, wedding and birthday at the other.

No single one of the 768 columns is "temperature," just as no single measurement on a doctor's chart is "fitness." But a particular combination of the columns (the hot-minus-cold arrow) behaves like a temperature gauge, and once you have the combination you can apply it to any row and read off a number.

The arrow trick even supports a kind of arithmetic that surprised the researchers who first noticed it. Take the row for king, subtract the row for man, add the row for woman, and look for the word whose row is closest to the result. In GPT-2's lookup table the answer is queen (similarity 0.71, with princess in second place). Do the same with bigger − big + small and you get smaller. The "male-to-female" direction and the "plain-to-comparative" direction are both real, reusable arrows in the space.

1.6 From a word to a whole sentence

So far, every word has a fixed row. But a word's meaning depends on its neighbors: bank in "river bank" is not bank in "bank account," and the sentence "The soup was cold" means something different from the sentence "The beer was cold." A fixed lookup table cannot capture that. This is where the rest of the model comes in.

The rows in the lookup table are only the model's starting point. As the model reads a sentence, it repeatedly rewrites the row of numbers sitting at each word position, mixing in information from the surrounding words, until the row at a given position no longer represents "the word was" but something more like "the state of this sentence so far, up to and including was." How that rewriting happens is the subject of Section 3 (layers). What it produces is a contextual encoding: a row of 768 numbers that encodes the sentence-so-far rather than a single word.

Two small housekeeping facts will help you read the later figures:

• The model does not quite work in words; it works in "tokens." Common words are one token each. Rare or long words are split into pieces. GPT-2 reads "Dear Sir or Madam" as five tokens: Dear␣Sir␣or␣Madam, and reads the word "unbelievable" as unbelievable. (The ␣ marks a token that begins with a space.) Punctuation marks are tokens too. When this article says "word," read "token."
• Every token position carries its own row of numbers. A ten-token sentence is, inside the model, a table of ten rows and 768 columns. Keep that table in mind: rows are tokens, and there are exactly as many rows as tokens. Section 3 shows how the layers rewrite all ten rows.

2How a language model is trained

Section 1 asked you to take on faith that the 768 columns get filled in "automatically during training." This section pays that debt. The punchline: training a language model is the line of best fit, with one change of task and a great many more knobs.

2.1 The line of best fit, revisited

You have seen a line fitted to a scatter of points. Recall what that actually involves. A line has two knobs: a slope and an intercept. For any setting of the two knobs, there is an error score: add up the squared vertical misses between the line and the points. Fitting the line means finding the knob settings that make the error score as small as possible. Figure 6 shows this in three panels.

Panel (c) deserves a second look, because it is the actual procedure used to train language models. For a straight line you could solve for the best knobs with a formula. With millions of knobs you cannot. Instead, you compute, for each knob, whether turning it up or down would reduce the error, turn each knob a tiny amount the helpful way, and repeat, thousands or millions of times. Each repetition is one adjustment step. The error creeps down. That is all "training" means: repeatedly nudging the knobs to shrink an error score. It is like tuning an old radio by ear: you cannot see the station, but you can hear the static, and you turn the dial a little at a time in whichever direction makes the static fade.

2.2 The task: guess the next word

A line of best fit predicts y from x. What does a language model predict? The answer is almost embarrassingly simple: given the text so far, it predicts the next word.

• "Peanut butter and ___"
• "Dear Sir or ___"
• "To be or not to ___"
• "The nurse checked the ___"

Rather than committing to one answer, the model produces a probability for every token it knows (50,257 of them for GPT-2). Figure 7 shows the real output for four prompts. After "To be or not to" the model puts 96% of its probability on be. After "Dear Sir or" it puts 68% on the token Mad (the first piece of "Madam"). After "The nurse checked the" it is spread out: patient, blood, baby, room, all plausible, none dominant. That spread is honest: many continuations are reasonable, and the model says so.

To produce a paragraph, the model simply repeats: predict a distribution over the next token, pick one (the most likely, or a random draw weighted by the probabilities), append it to the text, and predict again. Every fluent paragraph a chatbot has ever written was produced one token at a time in this way. Here, for instance, is GPT-2 continuing the prompt "The librarian opened the book and", picking randomly from its distribution at each step:

The librarian opened the book and found his hand pressed against the parchment. "I see," he said, "I know you're the one who writes and

It is not deep, and it stops mid-sentence because we stopped it after 25 tokens, but it is grammatical English on topic, produced by nothing more than 25 rounds of "guess the next token."

2.3 What plays the role of the error score

For the line, the error was the squared miss. For next-word prediction, the error is how little probability the model gave to the word that actually came next. If the true next word was jelly and the model had given jelly 90%, that is a small error; if it had given jelly 0.1%, that is a large one. (The precise formula is the negative logarithm of that probability, which behaves the way you would want: it is near zero when the model was confident and right, and it grows without bound as the model's probability for the truth shrinks toward zero.) Add this up over billions of words of text and you have the error score that training tries to shrink.

Now the whole recipe can be stated in one breath:

GPT-2 was trained this way on about 40 gigabytes of web pages (eight million documents). Modern models are trained on tens of trillions of words. Nobody labels anything. Nobody writes a rule. The text itself supplies the right answer at every position, because the right answer is simply the word that came next.

2.4 A tiny model, trained in front of you

It is hard to believe that "guess the next word" alone could produce the sensible word map of Figure 4. So here is a demonstration small enough to see all of, run for this article.

A made-up world has 36 words and 180 sentences, all of the form "<person> drank <drink> from a cup," "<person> ate <food> on a plate," "<person> drove <vehicle> to work," with four people, five drinks, five foods, five vehicles and a few variations of ending. A tiny next-word predictor was built with just two numbers per word instead of 768 (so the encoding can be drawn directly, with no squashing), plus one weighted-sum step from those two numbers to a probability for each possible next word. In all it has 180 knobs. The two numbers for every word started out random. Then the recipe above was run for 3,000 small adjustment steps. Figure 8 shows what happened.

Look at panel (d). All five drinks sit on top of one another. All five foods do. All four people do. The three verbs sit apart from everything, and apart from each other, because each verb predicts a different set of next words. The model was never told any of this. It was only told, 3,000 times, "your guess for the next word was too unsure; adjust." Words that play the same role in predicting what comes next get pushed to the same place, because putting them in the same place is the cheapest way to make the same predictions for them.

That is the entire secret of Figure 4. GPT-2's 768 numbers per word came from the same process, on a vastly larger scale: a much bigger world (the web), a much bigger lookup table, but the same rule. Words that are followed and preceded by similar things end up with similar rows.

2.5 What the knobs are

You do not need to know the model's inner wiring to follow this article, but one fact makes the next section much easier: almost every knob is a weight in a weighted sum. Figure 9 shows the whole idea with a miniature: rows of 5 numbers instead of 768, and a sentence of 3 tokens.

Read panel (a) first. A unit takes a row of numbers, multiplies each by its own weight, adds the products (plus a constant called a bias), and passes the total through a squashing function that keeps it in a sensible range. Out comes one number. If you have ever written a spreadsheet formula like = 0.3×A2 − 1.1×B2 + 0.05×C2 + …, you have built a unit; its coefficients are its knobs.

Now panel (b), which is where most people's mental picture goes wrong. One unit produces one number, so you might expect a row of 768 numbers to collapse into a single total. It does not, because the model does not use one unit; it uses a whole bank of them, side by side, each with its own set of weights, all reading the same input row at once. Five units produce five numbers; 768 units produce 768 numbers. The row is rewritten, not summed up: the output has exactly as many numbers as the input, each of them a different weighted combination of all the inputs. "Side by side" means nothing more than "many units, same input, one output number each."

Panel (c) puts the bank to work on a whole sentence. The table has one row per token. The same bank of units, with the same weights, is applied to every row separately: the row for The goes in and a new row for The comes out; the row for food goes in and a new row for food comes out; and so on. A sentence of ten tokens is a 10×768 table going in and a new 10×768 table coming out. Nothing gets shorter or narrower. At no point does the sentence become a single column of totals; the only place a row is ever boiled down to a handful of numbers is at the very end, when the final row is compared against every word in the vocabulary to produce the next-word probabilities of Figure 7.

Stack a few dozen of these operations one after another, each feeding the next, and you have a language model. This kind of arrangement, weighted sums stacked in stages, is what the term neural network refers to; the name is historical (the design was loosely inspired by brain cells) and carries no other meaning here. GPT-2's 124 million knobs are almost entirely the weights of its units, plus the 50,257×768 lookup table from Section 1.

There is one more ingredient, which lets a token's row look at the other rows in the sentence (it is called "attention," and it is why bank can be rewritten differently after river than after account). It, too, is built from weighted sums; the weights just get to depend on the words present. Section 3 shows it with real numbers.

2.6 After training, nothing moves

Training happens once, at enormous expense, and then stops. When you use the model, the knobs are frozen. Typing a prompt does not change any knob. What a prompt does is change the numbers flowing through the frozen machinery, the way a different set of inputs flows through the same spreadsheet formulas. Keep these two things separate in your mind:

• The knobs (weights). Fixed after training. This is the model's long-term knowledge.
• The activations. The rows of numbers sitting at each token position while the model reads a particular text. These change with every prompt. They are the model's short-term "state of mind" about the text in front of it.

How many rows are there, and which of them change? Suppose the model is given the four-token prompt "The soup was salty". It does not read the words one by one; it takes in all four tokens at once and creates four rows, one per token. Those four rows then go through the first stage of processing together, then the second, and so on (Section 3). When the model produces a fifth token and reads the extended text, a fifth row is created, and it turns out the first four rows come out exactly as before: the model is deliberately built so that a token's row can draw on the tokens before it but never on the tokens after it (we will see this in the mixing weights of Section 3), so adding words at the end never changes the rows of earlier words. In an experiment for this article, appending two tokens to "The food was" changed the existing three rows by less than 0.000002. So the picture is: one row per token, all present at once, earlier rows untouched by later ones.

Steering and conceptors, the subjects of Sections 4 and 5, never touch the knobs. They reach in and edit the activations, while the model is reading.

3Layers

Section 2 said the model rewrites the table of rows "a few dozen times, one stage after another." Each of those stages is a layer. This section shows what the layers do to the numbers, clears up the two most common confusions about them, and finds where in the sequence the meaning we care about is easiest to read.

3.1 Why stack simple steps

A single weighted sum can only do one simple thing: it draws a straight line (or its many-dimensional equivalent). That is why a line of best fit cannot follow a curve. The way to get complicated behavior out of simple parts is to stack them, so that the output of one stage becomes the input of the next. Each stage is simple; the sequence is not.

You have seen this structure many times. Grading an exam is a stack of stages: raw answers are turned into points per question; points are totaled into section scores; section scores are combined into a final grade. Each stage does a modest calculation, and the intermediate results (the section scores) are meaningful quantities in their own right, often more informative to a teacher than either the raw answers at the start or the single letter grade at the end. Hold on to that last observation; it will turn out to be true of language models too.

3.2 Rows are tokens; layers are versions

Here is the picture to hold in mind, and Figure 10 draws it with real numbers. The sentence is a table: one row per token, 768 columns. A layer takes that whole table in and puts a whole new table out, of exactly the same shape, with every number rewritten. GPT-2 has 12 layers, so there are 13 versions of the table: the starting version (rows copied straight from the lookup table of Section 1) and one version after each layer. Recent large models have around 60 to 120 layers, and correspondingly more versions.

Two confusions are worth heading off, because almost everyone falls into them.

Confusion 1: "the rows are the layers." They are not. Rows are tokens. A five-token prompt has five rows at every layer. The layers are not stacked inside the table like the layers of a cake; they are stacked in time, like successive edits of the same document. If it helps, think of thirteen photocopies of a spreadsheet, each a reworked version of the one before, all with the same rows and columns.

Confusion 2: "each row goes through all the layers on its own, then the next row does." Also not. All the rows go through layer 1 together, then all go through layer 2 together, and so on. This matters because, inside a layer, a row is allowed to look at the other rows (that is how context gets in). At the end, the model reads its next-word prediction off the last row of the final table, the row for the most recent token; that row has had twelve chances to absorb everything before it.

Figure 11 opens up one layer to show the three steps by which it turns one version of the table into the next.

So, in plain words, a layer does three things to the table:

1. Mixing. Each row is replaced by a blend of itself and the rows before it (the row for was becomes mostly The plus a little food plus some of itself, in the example). The blend is still a row of 768 numbers. The blend weights are computed on the spot from the words present; they are not fixed knobs, which is what lets the same layer treat bank differently after river than after account. (The model actually runs twelve such blends in parallel, called "attention heads," each with its own weights; the figure shows their average.)
2. The bank of weighted sums. Each blended row goes through the bank of units from Figure 9, separately, producing an update: a row of 768 numbers.
3. Write-back. The update is added to the row. "Writes back" simply means that the layer's output is old row + update, for every row of the table. Because it is an addition rather than a replacement, information laid down by early layers survives to the end. (The research literature calls each row, as it is carried through the layers and added to at every one, the residual stream; whenever you meet that phrase in a paper, or in this article's appendix and glossary, it means exactly this: the row of numbers for one token, as it passes through the layers.)

One consequence of the write-back is visible in the numbers of Figure 10: because every layer adds to the row, the numbers in it get larger with depth. Figure 12 shows the typical size of one number in the row for was: about 0.2 in the lookup-table version, about 2 after layer 1, and about 18 after layer 12, a roughly hundredfold growth. This is a mundane fact, but it matters in Section 4, because "how big a nudge is big" depends on which version of the table you nudge.

What is the model doing as it rewrites the table? One crude but revealing trick is to take the row for was at an intermediate layer, pretend it is the final version, and ask what next-word prediction it would produce. (Researchers call this a "logit lens.") Doing that for the same prompt gives the table below.

Read it top to bottom as a sequence of drafts. At the start the row knows only that the word is was, and the guesses are word-fragments. After one layer it has absorbed enough context to propose grammatical continuations (not, also). Through the middle layers it develops a view about what kind of thing comes next (an evaluation of food: delicious first appears at layer 8 and dominates by layer 9). The final layer spreads the probability over the specific plausible words (delicious, good, so, very). The topic ("food") and the tone ("positive") are present in the middle versions before the exact word is chosen at the end.

3.3 Where does "positive versus negative" live?

The most important experiment in this article is a simple one, and it uses only the two-group comparison behind a t-test. Forty-eight pairs of short sentences were written for this article, each pair identical in topic and structure but opposite in sentiment:

Each sentence was fed to GPT-2. A sentence of ten tokens gives a table of ten rows at every layer, and for this experiment we keep just one of them: the row at the sentence's final period. From here on that row is called the sentence's summary row, and there is one summary row per sentence per layer. Why the period? Because at the start, the period's row is identical for every sentence (it is just the token "." from the lookup table). Whatever the period's row knows about the sentence at later layers, it must have picked up from the layers, by mixing with the rows before it. That makes it a clean probe of what the layers write.

Now the comparison. At each layer we have 48 summary rows from positive sentences and 48 from negative ones, each row 768 numbers long. Ask the question a t-test asks: how far apart are the two groups, relative to their spread? Figure 13 shows the arithmetic with the literal numbers, and it is worth walking through, because the same three steps recur throughout the rest of the article.

1. Average. Add up the 48 positive summary rows, column by column, and divide by 48: one average row. Do the same for the 48 negative rows. (Panel (a), the two bold rows.)
2. Subtract. Positive average minus negative average, column by column. The result is a row of 768 numbers, the arrow. It is exactly what the numerator of a two-sample t-test would be, computed for each of the 768 columns and collected into a row. (Panel (a), bottom row.)
3. Project. To give one sentence a single score, multiply its summary row by the arrow, column by column, and add up all 768 products. (Panel (b).) Geometrically this measures how far along the arrow the sentence sits; a sentence whose row points the same way as the arrow gets a large positive score, one pointing the opposite way a large negative score.

With one score per sentence, the rest is the familiar two-group comparison: take the gap between the two groups' mean scores and divide by their pooled standard deviation. That ratio is Cohen's d, the standardized mean difference. At layer 8 the positive sentences average +10.1, the negatives −10.1, the pooled spread is about 3.0, and d = 20.1 ÷ 2.95 = 6.8 (panel (c)). Figure 14(a) repeats this calculation at every layer. One caution: the arrow was computed from the same 96 sentences it is then used to score, which flatters it a little. The dashed line in the figure redoes it properly, computing the arrow from 36 of the pairs and scoring only the 12 held-out pairs (averaged over twenty random splits); it runs just below the solid line (6.7 against 6.9 at layer 8), so the flattery is small, and panel (b) is held out throughout.

Three features of the curve deserve attention.

1. It starts at zero. At the start, straight from the lookup table, the period's row carries no sentiment, as promised; the two groups are indistinguishable (d = 0.06, and the sort-by-nearest-average test is at 52%, a coin flip). The tokens have no "meaning" in the model until they have run through the layers.
2. It peaks in the middle. By layers 7 and 8 the two groups are almost seven standard deviations apart. For comparison, a "large" effect in a psychology experiment is d = 0.8. The model has written "this sentence was positive/negative" into the summary row so firmly that a rule as crude as "which average is closer?" sorts 100% of held-out sentences correctly.
3. It drops at the end. After layer 12 the separation has fallen to d = 3.6. The last layers are busy converting meaning into a specific next-token guess, and in doing so they partly overwrite the tidy summary that the middle layers built.

Figure 15 shows the same story as pictures. At each of four depths, the 96 sentences are plotted with the horizontal axis being the arrow (computed from 36 of the pairs) and the vertical axis being the largest other direction of variation; the 12 held-out pairs are drawn hollow. After layer 1 the groups overlap; after layer 8 they are cleanly split, held-out sentences included; after layer 12 they have begun to merge again.

3.4 This is a general finding, not a quirk of one experiment

The "middle layers hold the most readable meaning" pattern shows up throughout the research literature, on models far larger than GPT-2:

• Skean and colleagues (2025) tested every layer of several models as a source of sentence encodings on 32 standard tasks and found that an intermediate layer beat the final layer in nearly every case, by up to 16 percentage points, in models of several different designs. Their explanation is that the middle layers keep the essential information while discarding low-level clutter, and the last layer is "overly specialized" to the next-word guess.
• Zhu, Subramanian and colleagues (2026) did a version of our experiment with three groups instead of two. They fed descriptions of depression symptoms to a 62-layer model, grouped into mood symptoms, bodily symptoms and thoughts of death, and asked at every layer how far apart the three groups' summary rows sat relative to their spread (a three-group version of Cohen's d; the groups need not be opposites, any more than three treatment arms in a trial need to be). Three different distance measures agreed on the same answer: the groups were most separable at layer 21, about a third of the way in.
• Ravenda and colleagues (2025), steering a 32-layer model, simply chose the middle layer, citing exactly this body of evidence.
• Triantafyllopoulos and colleagues (in preparation), the central source on conceptors, read their numbers from the midpoint layer of each of five models (layer 13 of 26, 21 of 42, 18 of 36, 16 of 32, 16 of 32).
• Gurnee, Sofroniew, Lindsey and colleagues (2026) report that the concepts a model is "poised to talk about" carry coherent content only in an intermediate band of layers, with the last layers tied to the imminent output.

4Reading and steering

Section 3 found that, at layer 8, the summary rows of positive and negative sentences sit far apart along one arrow. This section uses that arrow twice: first to read the model's assessment of new text, then to change it. The second use is what "steering" means.

4.1 The arrow, stated plainly

You have now met the arrow several times, so here is its definition in one place. Take a set of examples of one kind (positive sentences) and a set of the opposite kind (negative sentences). Run them through the model. At a chosen layer, take each sentence's summary row (here, the row at its final period). Average the rows within each group. Subtract:

It is a row of 768 numbers (Figure 13(a), bottom), and it is exactly the numerator of a two-sample t-test, computed separately for each of the 768 columns and collected into a row. In the research literature it is called a "difference-of-means direction," a "mass-mean probe," or, when used for steering, a "steering vector" or "contrastive activation addition." All of these are the same object. Note that it always takes two groups to make one: the arrow points from the second group's average toward the first group's average, so it has a direction; averaging the two groups together would give you a center point, not an arrow.

4.2 Reading: a ruler for sentiment

Given the arrow, any new text can be scored: run it through the model, take its layer-8 summary row, and ask how far along the arrow it sits (measured from the midpoint between the two group averages; this is step 3 of Figure 13). Figure 16 does this for sentences the arrow never saw: twelve held-out positive sentences, their twelve negative twins, sixteen neutral sentences with no opinion in them, and ten brand-new sentences written afterwards specifically to try to trip it up.

Nothing in this ruler was trained. No classifier was fitted. Two averages were subtracted, and the resulting arrow generalizes to sentences about mechanics, dentists and rent that have nothing to do with the movies, hotels and coffee it was built from. That generalization is the evidence that the arrow captures something like "positive versus negative" itself, rather than "these particular 72 sentences."

The same recipe reads other things. Marks and Tegmark (2024) built datasets of simple true and false statements ("The city of Paris is in France," "The city of Hanoi is in Poland," "Sixty-one is larger than seventy-four") and computed true-minus-false arrows inside the Llama-2 models (a family of openly released models from Meta, in sizes from 7 to 70 billion knobs, some sixty to six hundred times larger than GPT-2). In the two larger models, an arrow built only from number-comparison statements sorted true from false statements about Spanish vocabulary with better than 95% accuracy: same arrow, unrelated topic.

A concept does not have to come in opposite pairs to be read this way. Zhu, Subramanian and colleagues (2026) worked with three groups of depression-symptom descriptions taken from the wording of clinical questionnaires: mood symptoms ("feels sad most of the day"), bodily symptoms ("wakes in the early hours and cannot fall asleep again") and thoughts of death. None of the three is the opposite of another. So instead of one arrow between two groups, they used the three group averages as three landmarks in the number-space (a row in their model has 5,376 numbers rather than 768; nothing else changes), and scored a new passage by how far it lies toward each landmark, adjusting for the fact that the three landmarks are not at right angles to one another. When they scored passages from a memoir and a clinical handbook, passages that clinicians had labeled "bodily" scored highest on the bodily landmark, and likewise for the other two. They did also build one genuine arrow, from all three symptom groups together toward a group of cheerful passages, and it separated depressive from happy text on held-out data. Same arithmetic, arrows or landmarks as the concept requires.

4.3 Steering: pushing the numbers

Now the leap. If the model's later layers read sentiment off the layer-8 row, what happens if we write to that row? Concretely: let the model process a prompt as usual up through layer 8; then, before layer 9 runs, add the arrow (multiplied by a chosen strength) to every row of the table; then let layers 9 through 12 continue as if nothing had happened. None of the model's knobs change. Only the activations, the numbers passing through, are edited. This is steering.

Figure 17 shows the result for the prompt "We tried the new restaurant on Main Street. The food was really ___". With no steering, the model gives 74% to good and a further 15% to other positive words; it gives essentially nothing to negative words. Add the arrow with strength −2 and the picture flips: 65% bad, then awful, terrible, poor; positive words fall to about 1%. Add it with strength +1 and the positive words become more varied and enthusiastic (great rises from 5% to 11%). The strength is a dial, and the model's mood turns with it.

The edit changes not just the next word but everything the model goes on to write, because each generated token is fed back in and processed through the same edited layer. The table below shows full continuations (20 tokens, always taking the most likely token) at several strengths, for three prompts.

Notice that the arrow was built from sentences about movies, hotels, coffee and gardens, yet it steers a paragraph about a phone. Notice also the repetitiveness: GPT-2 small is prone to loops even unsteered ("I was really disappointed. I was really disappointed."), and steering does not cure that. And notice what happens at strength +5 and +10: the text degenerates into "very very very" and "and and and." Pushing a row of numbers far outside the region the model has ever seen does not produce a very, very positive review; it produces nonsense. Steering is a dial with a useful range, not a slider to infinity.

4.4 Steering is not prompting

It is worth being precise about the difference, because Section 6 turns on it.

• A prompt is more text. "Please write a positive review." The model reads it the way it reads everything, through the lookup table and all twelve layers. Whatever effect it has, it has by way of the model's ordinary reading process, and the model's ordinary reading process can misunderstand it, weigh it lightly, forget it after a few hundred words, or be talked out of it by later text.
• Steering is an edit to the activations. It does not pass through the reading process. It is applied at a chosen layer, at every step of generation, with a chosen strength, regardless of what the text says. It cannot be "misread," because it was never read.

Neither one changes the model's knobs. Both leave the model exactly as it was for the next user. But they act at different places: one at the input, one in the middle of the computation.

4.5 A real example: steering a model to correct a cautious bias

Ravenda and colleagues (2025) faced a practical problem. They wanted a small model (Llama-3.1-8B, an 8-billion-knob open model) to judge whether Reddit posts were relevant to particular items of a standard depression questionnaire (the BDI-II). Small models are attractive here for privacy and cost. But the small model had a systematic cautious bias: it flagged far too many posts as relevant. The authors describe this as a preference for avoiding false negatives in a sensitive clinical setting; where that preference came from, they do not say, and neither will this article.

So they built an arrow. For each questionnaire item, they collected the model's middle-layer rows when it answered correctly and when it answered incorrectly, and took the difference of the averages. Then they steered with it, choosing the strength by a calibration rule on validation data. Figure 18 shows their published results on the test set.

Two lessons. First, the correction was applied inside the computation rather than by rewording the question: the authors did not report trying to fix the bias with a different prompt, so the paper does not show that a prompt would have failed; what it shows is that an edit to the middle-layer numbers, built from two averages, was enough. Second, the strength matters, and there is a real trade-off; the same figure shows that over-steering destroys the sensitivity that made the model useful. The authors went on to use the steered model to fill in whole questionnaires from users' posting histories, and reported that the steered 8-billion-knob model was competitive with much larger commercial models on that task, ahead of some and behind others.

A second published example points the other way, toward reading rather than fixing. Abootorabi and colleagues (2026) built steering arrows for twenty human values (benevolence, achievement, tradition, self-direction, …) in several models, and asked whether the angles between the arrows matched the well-known psychological "circle of values," in which compatible values sit near each other and conflicting ones sit opposite. For arrows built the simple difference-of-means way, they did: the correlation between the model's arrow geometry and the psychological theory reached 0.5, and steering a model toward one value nudged it toward compatible values and away from opposing ones, as the theory predicts. The arrows are not arbitrary; they reflect a structure the model has learned about people.

4.6 The limits of a single arrow

The arrow is a wonderful tool, and it will not go away in Section 5; conceptors are built on top of the same material. But it has three limitations, and naming them now will make conceptors feel inevitable rather than arbitrary.

1. Everyone gets the same push. Adding the arrow moves every row by the same amount in the same direction, whether the sentence was already strongly positive, weakly positive, or negative. A review that was already glowing gets pushed toward "very very very"; a review that was scathing gets pushed only part-way toward neutral. A push that does not depend on where you started is a blunt instrument.
2. It is one direction. A concept like "negative sentiment" is not a single line. There is anger, disappointment, boredom, disgust; there is the difference between a bad meal and a bad movie. In the 768-number space, "the set of negative sentences" is a cloud with a shape, and the arrow only records where its center is relative to the positive cloud's center. It records nothing about the cloud's spread.
3. It is estimated from averages, so it is noisy when examples are few. With two or three examples per side, the arrow is mostly the idiosyncrasies of those examples. Nothing in the arrow itself knows which of its 768 components are "sentiment-like" and which are noise.

A conceptor addresses all three. It keeps the arrow's information about where the concept sits, adds information about the concept's shape, uses that shape to act differently on different inputs, and uses it to clean up noisy estimates. That is the next section.

5Conceptors

A steering arrow records one direction: from the average of one group of examples to the average of its opposite. A conceptor records, in addition, the shape of the region the examples occupy, and uses that shape as a filter. This section builds a conceptor from the same 96 sentences as before, one step at a time, keeps returning to actual sentences to show what each step means, and shows what a conceptor does that the arrow cannot.

5.1 A concept has a shape

Go back to the 96 summary rows at layer 8 (one row per sentence: the row at its final period), and look at them from a new angle. Figure 19 plots them with the arrow as the horizontal axis (as in Figure 15) and, vertically, the direction along which the sentences vary most apart from the arrow. Each dot is one sentence's summary row, placed by two of its coordinates. Ellipses are drawn around each group and around all 96 together.

Two observations, both of which will matter.

First, each group is a cloud with a shape. The positive sentences do not all sit at one point; they spread out, and the spread has structure: it is much larger in some directions than in others. Some of that spread is meaningful variation within the concept (a glowing review of a concert is not the same thing as a glowing review of a laptop); some is topic and wording that has nothing to do with sentiment.

Second, and this surprises most people, the biggest direction of variation among all 96 sentences is not the arrow. The standard deviation of the pooled cloud along the arrow is 10.5; along the vertical direction in the figure it is 11.8. To say this precisely we need a ranking that the next subsection introduces properly: take the pooled cloud and list its directions of spread from the one it stretches along most to the one it stretches along least (a list we will call direction 1, direction 2, and so on). The arrow lines up best with direction 3 on that list, not direction 1. Directions 1 and 2 are mixtures of sentence shape and sentiment that both groups share. If you tried to discover "sentiment" by looking for the biggest direction of variation in the data (the way principal component analysis would), you would find something else first. This is why the arrow is computed from the group averages, and why a conceptor, which captures the whole shape, needs the labels too.

5.2 From spread-per-direction to gain-per-direction

Here is the recipe for building a conceptor from a set of example summary rows. One word first, because it is used throughout: a gain is a multiplier applied to a signal, the way the volume knob on an amplifier multiplies a sound. Gain 1 means "pass it through unchanged," gain 0 means "silence it," gain 0.3 means "turn it down to 30%." It is a neutral word; a high gain is not "good," it just means "kept." A conceptor is a set of gains, one per direction, and for that reason this article will often call it a filter: something you pass a row through, that keeps some of it and turns down the rest.

1. Center the examples. Subtract the average row from every example, so the cloud sits around zero.
2. Find the directions of spread and how much spread there is along each. Any cloud of points can be described by a set of perpendicular directions, ordered from "the direction the cloud stretches along most" to "the direction it stretches along least," with a variance attached to each. (For a two-dimensional cloud, these are the long and short axes of the ellipse that best fits it; in 768 dimensions there are 768 such directions. This is the same idea as principal component analysis.) Figure 20(a) shows the variances for our 96 sentences: the biggest direction has variance 142, the tenth about 27, the fortieth about 4, and the ninety-fifth 0.26. With only 96 examples, a cloud can only spread in at most 95 directions, so the remaining 673 of the 768 directions have variance exactly zero.
3. Turn each variance into a gain between 0 and 1. This is the one formula in the article, and it is a very tame one:
   gain along a direction  =  spread  ÷  (spread + λ)
   where λ ("lambda") is a single positive number you choose (more on it below). If the spread is much bigger than λ, the gain is close to 1. If the spread is much smaller than λ, the gain is close to 0. If the spread is exactly zero, the gain is exactly zero. Figure 20(b) shows the curve and the actual gains for our 95 directions at λ = 1.4: the top direction gets 0.99, the fortieth gets 0.73, the sixtieth 0.50, the ninety-fifth 0.16, and the 673 directions with no spread get 0.
4. The conceptor is the list of directions together with their gains. That is all it is. Written as a formula in the research literature, the conceptor is C = R (R + λI)⁻¹, where R is the covariance matrix of the centered examples; but that formula does nothing more than steps 2 and 3 at once, and you can read it as "the covariance, with each direction's variance replaced by its gain."

What do these directions actually correspond to, in sentences? This is the natural question, and the honest answer is: each direction is an axis with two ends, not a category. "Positive" and "negative" are not two different directions; they are the two ends of one direction. To see which is which, take each direction in turn and look at which sentences sit at its two extremes. For our conceptor ("arrow overlap" is how much the direction lines up with the sentiment arrow: 1 would mean identical, 0 unrelated):

Direction 3 is the sentiment axis: negative sentences at one end, positive at the other. Direction 4 runs from first-person feelings ("I feel …", "I love/hate …") to remarks about books, paintings and songs, with both positive and negative sentences at each end: it is a topic axis, not a sentiment axis. Direction 5 runs from sentences about people (teachers, doctors, a presenter) to sentences about places and weather, again with both sentiments at each end. Directions 1 and 2 (not shown) mix sentence shape with sentiment. So "the 20 strongest directions" of this conceptor means: the sentiment axis, plus a couple of shape-and-sentiment axes, plus a collection of topic axes along which the example sentences happen to vary. A conceptor that keeps all of them keeps "the kinds of variation that occur among opinionated sentences like these," which is both more than the arrow and, as Section 5.7 shows, sometimes more than you wanted. One point of bookkeeping, since the phrase recurs: the experiments below use a conceptor cut down to those 20 leading directions, with their gains, and the other 75 directions dropped. That is a truncated conceptor, not the full one defined in step 4, which keeps all 95 directions; the cut is made to match the protocol of Triantafyllopoulos and colleagues, who use 20 throughout, and the appendix reports that the full conceptor gives the same results in every experiment where both were run.

To apply a conceptor to any row of numbers (the summary row of a new sentence, say): express the row as a combination of the conceptor's directions, multiply the amount along each direction by that direction's gain, and put the pieces back together. Directions the examples spread along a lot pass through almost unchanged; directions the examples never used are removed. Figure 20(c) shows the effect on inputs of every direction: a circle becomes an ellipse whose long axis is the direction with the highest gain.

5.3 The aperture: the "how picky" dial

λ controls how sharply the conceptor discriminates between well-used and rarely-used directions. The research literature usually talks instead about the aperture, a number α with λ = 1/α², so that a wide aperture (large α, small λ) lets nearly every direction through, and a narrow aperture (small α, large λ) keeps only the dominant few. Figure 20(b) shows the curves for four settings. Wide aperture: almost all 95 directions get gains near 1 and the conceptor is close to a plain "keep everything the examples touched." Narrow aperture: only the top handful survive. The experiments in this article use λ equal to the average variance per direction (about 1.4 here), the default used by Triantafyllopoulos and colleagues. The average gain across all 768 directions, called the quota, is 0.076 for our conceptor: it keeps a small slice of the space and mutes the rest, which is what a filter for one concept should do.

5.4 Three things you can do with a conceptor

Before the figure, the correspondence to keep in mind, because it is easy to lose: a row (768 numbers) is drawn as a dot; a set of example sentences is a cloud of dots; the cloud's shape is the ellipse the conceptor records; the arrow is the line from one group's center to the other's. Figure 21 draws all of this in two dimensions instead of 768 (so that the dots can be seen), on a toy cloud, and shows the three operations that come up in the research literature, next to the arrow for comparison. Six test points are marked; think of each as a new sentence's summary row.