Steal the Shape, Not the Mechanism

Last week Moonshot published Attention Residuals. It’s a change to the inside of a transformer, which is nowhere near where I work. I build Emil, an LLM that lives on a Minecraft server, and I never touch the model’s layers. I read the paper anyway, because I’ve started reading papers for their shape rather than their mechanism, and this one had a shape I recognised.

The mechanism is this. A transformer’s residual stream is a running sum: every layer’s output gets added once, with weight one, and stays added. Layer fifty sees all forty-nine predecessors equally. Moonshot replaces the sum with attention over depth, so each layer picks which earlier outputs it reads. The sum stops growing without bound, nothing gets diluted by fifty siblings, and they report it’s worth about a quarter more training compute.

Written out, the two look like this:

residual stream      h_l = v_1 + v_2 + … + v_(l-1)            every weight is 1

attention residuals  h_l = Σ α_i · v_i,  α = softmax(w_l · norm(v_i))
                                          ↑         ↑
                              weights sum to 1     layer l's own question

Each layer carries a small learned vector w_l, a question it asks of every earlier output. Outputs that answer it get weight, the rest fade. Two properties fall out: the weights sum to one, so the stream can’t grow without bound no matter how deep the net is, and every v_i has the same width, so attending over a hundred of them costs the same as over ten.

The shape is: replace an accumulation with fixed unit weights by one where the current state chooses. Written that way, it stopped being about transformers.

Emil has a residual stream

Emil’s context window is a running sum. Every turn appends an observation, a tool call and a result, each with weight one, and when the window fills, the oldest goes. The crafting table he placed forty turns ago is gone because thirty-eight walk acknowledgements were appended on top of it. Same complaint, with a turn index in place of the layer index.

So I asked the coding agent whether we could test the same idea one level up. Its answer was that the mechanism has nowhere to go, there are no gradients in a harness, but the diagnosis maps, and it pointed at the three places in the code where history gets cut. Two of them are the same operation: drop from the front. Then I had it explain the equation to me properly, because I wanted to know what I was borrowing.

Dedre Gentner described the borrowing in 1983. What transfers between domains in a good analogy is relational structure, not the attributes of the objects. The atom is like the solar system because something small orbits something large under a central force, not because electrons are yellow. “Learned weights instead of unit weights” is a relation; “softmax over RMS-normalised hidden states” is a colour. The relation made the trip. The colour stayed in the paper. Hold that thought about norm.

Mapping it

First, a count. I had assumed the problem was overflow: the context fills up and the harness resets it. That’s the problem the paper’s bounded-sum result solves, so that’s where I looked. The agent counted what actually happens to Emil’s window over the last 2,600 live turns:

window operation	count
reset (context full)	0
evict	1
trim oldest	78

Emil never overflows. He forgets gradually, oldest first, to a fixed horizon of about half an hour. So the experiment isn’t “survive overflow better”. It’s: same budget, but the newest observation decides which old turns stay. A count is cheap, and it picked the right problem before any code existed.

Second, the query. The paper’s w_l is a question each layer asks. Emil already had one and I hadn’t noticed: his diary recall builds a search string from the current observation — held item, inventory, next plan step, a recent death — and scores diary entries against it. That string is the query. Run the same scorer over old turns instead of diary entries, keep the top few. The paper keeps a block of recent layers as a plain sum and attends only across blocks; Emil keeps the newest six turns verbatim and selects four among the older ones.

Third, what not to reuse. Months ago I’d shelved a context compressor: a small model that scores every token for keep-probability and drops the low ones. Tempting to bring it back. The agent read the crate and said no, for two reasons. It has no query — it scores tokens on how much they look worth keeping, which is a prior, a fixed mask, the one thing the paper adds being the one thing it lacks. And it produces text Emil never saw: mask “need 3 have 2” out of a failed craft and what’s left reads like a success, which is the bug I’ve spent the most time on. So old observations get a structural digest instead — a pure function that keeps the durable fields and drops the rest — and anything dropped entirely stays reachable through the diary Emil writes himself.

The shape had four parts

I’d borrowed one relation from the paper. The paper has four. Two showed up while building, the last one in the trace.

1. The current state chooses. Score old turns against the query the newest observation builds, keep the top few. One detail transferred with it: attention scores keys, not the whole hidden state. Scoring whole turns lost to first-in-first-out, because every observation lists the inventory and so does the query. Scoring only what Emil did and what came back, the key, wins.

2. Attend over everything, every layer. The paper never drops an earlier output; each layer re-picks from all of them. My first version only picked among turns still in the window, and the same four held their seats for two hundred turns. So dropped turns go to a pool, and the top four are re-picked each turn from pool and window together. A turn that left can come back.

3. Fixed-size values. Every layer’s output is the same width; attention never has to pay for a big one. Emil’s old observations weren’t, and the window grew. Now a kept observation shrinks to held item, inventory, health, next step, last death, and results are cut at a fixed length with the cut stated in the text.

Four recorded episodes, replayed through both policies at the same budget. “Hit” is the share of turns where the kept old context contains a term the current query asks for:

That looked like a win for about an hour, until I asked for the traces.

4. Normalised keys. 1,937 restores over the iron-age run, 669 distinct turns. The top one came back 277 times. It’s turn 16:

query    [next] 1. Walk south to the birch tree cluster · Beach · night ·
         CookedChicken x3 · Bow x1 Diorite x2 Stone x10

turn 16  chat("i'm alive again, i think. low on health and no tools so i'm
         heading to find some trees first to make a pickaxe. saw stone all
         around here but nothing i can break with it.")

Trees, pickaxe, stone, health. A chat line that happens to name half the inventory out-scores everything for hundreds of turns. Turn 88, “just died again to a skeleton lol, getting wood now”, came back 257 times. And at the moment this whole thing was for, plan step “craft furnace, smelt 3 ingots”, raw iron and coal in the inventory, what gets restored is:

turn 807  set_plan({"steps":[{"text":"Get birch wood from nearby trees", …
turn 813  remember({"name":"current goal","body":"Finding iron ore to get
          iron pickaxe. At (57,23,-37) in LushCaves …
          → {"ok":false,"note":"no memory called \"current goal\""}

Two plan edits and a failed memory write, because they’re long and full of item names. The furnace placement, a short call with a short ok, never scores.

So the 71% is real and it’s measuring the wrong thing: word overlap with whatever old turn talks the most. That’s the norm I filed under colour in the morning. Keys are normalised so that no output wins by being large, and it isn’t decoration, it’s the fourth relation. What the text version of it looks like, scoring the call alone, or a length penalty, or only turns that touched a block, is tomorrow’s ruling.

Where it stands

	before	after
selection	oldest goes first	newest observation asks, best 4 old turns stay
scope	window only	window + pool of everything dropped
values	full observation, full results	digest + capped results
recorded as	trim	drop / restore, replayed byte-for-byte
cost	none	one lexical scoring pass per turn, no GPU
scorer	—	wrong: rewards length, not relevance

Emil is running tonight on the old policy. Nothing from the new one goes live until the scorer can tell a furnace from a chat about trees.

The pattern was four relations long and I’d read it as one. Two I found by building, one by reading the trace instead of the number. Next paper, I’ll count them before I start, and I’ll be slower to call anything a colour.