Giving an LLM Eyes: a Low-Res Snapshot of the World

How do you make a language model see? It’s an open problem people are attacking from every side right now, because agents keep leaving the chat box for worlds with geometry in them — and geometry is where models still fail. Even vision-language models routinely misjudge the basics: distance, counting, what is in front of what (arXiv:2407.06581).

The failure is structural. A vision-language model doesn’t see a picture: the image is downscaled, cut into a coarse grid of patches, each patch becomes a learned vector trained to capture what’s in it, and the language model attends over those vectors like exotic words it has to translate back into language. Identity survives that pipeline; geometry mostly doesn’t. There is no depth channel, position is a learned hint rather than a coordinate, and anything smaller than a patch smears.

For Emil the question is sharper still: he is a text-only model, so his sight has to arrive as text. Since August 10th it does, in one piece: every turn, the observation carries a low-resolution snapshot of the world around him — a full-sphere panorama, rendered as a matrix. No look command, no tool call. He sees the way you do when you open your eyes: all the time, in every direction, at a resolution that fades with distance.

At the bottom of this post you can watch the actual matrix from the live server, the same bytes the model reads.

The matrix

Columns are compass bearing: 48 of them, 7.5° apart, column 1 pointing north and wrapping east–south–west. Rows are pitch: 35 of them, 5° apart, from +85° down to −85°, plus one cell each for straight up and straight down. That is 1,682 lines of sight. Each ray marches voxel by voxel from his eyes out to 32 meters, and the cell records what it reached:

To10n-17.18         a torch, 10 m out, its north face visible, cell 17.18
Wil6w/Cra18w-12.18  a wildflower — and seen THROUGH it, a crafting table
.                   sky, or nothing within 32 m

Every token has the same anatomy. Taking the middle one apart:

That cell is real, from a night in a birch forest last week. Rays pass through non-occluding blocks like leaves, water, and glass, reporting each layer before the surface that stops them. A legend line above the grid maps the kind codes to block names, and the codes are fixed for good: St is Stone in every frame Emil will ever see. Drawn the way the live viewer at the bottom of this post draws frames — one pixel per line of sight, colored by kind, darker is farther — the whole frame that cell came from looks like this:

Three rules keep the render honest:

• Occlusion is real. A ray stops at the first solid block, and what is behind that surface stays unknown until he digs. The panorama and the tools share one voxel traversal, so nothing seen ever bypasses the no-wallhacks rule from the first post.
• Unknown is unknown. An unloaded chunk stops a ray and reports nothing, rather than pretending to be sky.
• Resolution fades with distance. At 32 meters adjacent rays are about 4 blocks apart, so small distant things fall between them. That is a fact about eyesight, not a bug — going closer is his move, the same as in the game. Past 32 meters, walking is looking: the panorama re-renders as he moves.

In sensor terms, the matrix is a lidar scan. A spinning lidar samples the world the same way — beams at fixed elevations, stepped around in azimuth, one range per direction — and robotics calls the resulting grid a range image. The quirks carry over too: real lidar also gets multiple returns through foliage, the way a cell here chains a leaf in front of a log, and its small distant objects also fall between diverging beams. The difference is that a lidar return says something is here and the rest of the stack has to infer what; every return here arrives labeled, because the sensor is the same process that renders the world. Driving simulators ship exactly this idealized instrument — CARLA calls it semantic lidar — because when you own the world, there is no reason to make perception guess.

And in model terms, there is a neat symmetry hiding here. A vision-language model’s retina is also a low-res grid: a 336-pixel image enters the model as roughly 24×24 patch vectors, about the same cell count as this matrix. The difference is what a cell carries. A patch embedding is a fuzzy learned summary the language side has to decode on faith; a matrix cell is verified identity, metric depth, and a handle the model can act on. The panorama is what the VLM pipeline would produce if every stage were lossless and the output were already words.

Seeing and acting are the same token

The trailing 17.18 in a cell is its id, and ids are handles. The model never translates what it sees into coordinates; it acts with the token as shown: hit{target:"To10n-17.18"}, goto{target:"G2u-24.20"}. The harness resolves the id back to the block and checks the kind code on the way, so a stale frame can never silently hit a different block than the one named. The distance in the cell is measured from his feet — the same number the tools compare against — so what a token says predicts what the tools will do with it.

One line under the legend covers the case sight alone can’t: standing in a tunnel, every cell is a wall a meter or two away, and nothing in the whole matrix is a legal place to walk. The open line answers it directly — eight bearings, how far standable floor runs each way, and the coordinates that walk would end at, ready to paste into goto.

With the matrix in place the observation got simpler, not busier. A turn now carries his own state, his plan, the matrix, and what he hears. The hand-picked “nearby” lists from the first post are gone; the snapshot replaced them.

What a look costs

Sight is by far the biggest single thing in Emil’s context. Measured with the served tokenizer over 60 live turns: a panorama is a median of 19,000 characters that tokenize to a median of 14,600 tokens. The direction of the range is the opposite of what you might guess: open terrain is the cheap case, around 9,000 tokens, because a sky cell is a single dot and a forest view is hundreds of them. The expensive frames are underground and underwater, up to 20,000 tokens, where all 1,682 rays hit something and every cell is a full token bundle. The matrix is dense symbol soup — G6u-23.20 — so it packs only about 1.5 characters into a token where ordinary prose gets 4. The rest of the observation is a rounding error next to it: the full turn lands around 15,000 tokens, of which the matrix is 97%.

But that is occupancy, not spending — and the distinction is the whole design. The compression rule for old panoramas is blunt: sight is current-frame only. The moment a new observation lands, every older turn’s see field is stripped out of the conversation history, so exactly one frame is ever resident, paid once as rent rather than again each turn. Measured on the live session as I write this: the full prompt he is served — system prompt, history, current observation — is 37,000 tokens, and the resident frame is 15,100 of them, 41% of everything he reads. What a turn adds to the context for good is tiny. A stripped past turn settles to about a thousand characters — 364 tokens for a typical one, by the same tokenizer — and across eight turns of the episode running as I write this, the harness’s own accounting shows the context growing by roughly 240 tokens a turn: the trimmed observation, his reasoning, his tool calls and their results. Sight is his most expensive resident and his cheapest habit.

That 41% deserves a denominator check, because there are three windows in this system and they disagree. The model itself is served with room for 65,536 tokens, so one frame occupies 23% of what it could physically hold — the ceiling is a choice, not the hardware. The harness holds the prompt to its own budget of 36,000, but it counts in a blind estimate of four characters per token, and the matrix packs 1.5 — so the guard books a frame at about 5,100 tokens, 14% of budget, a threefold undercount of the real thing. In the first live days that gap let a history holding several frames sail straight past the guard into the serve window. Current-frame-only stripping closed that class of failure structurally: however wrong the estimate, there is only ever one frame to be wrong about.

The stripping isn’t only about budget. A stale frame is noise: it describes where he stood, not where he stands, and a model reasoning over last minute’s walls walks into this minute’s. What survives from a past look is what he did about it — plus whatever he chose to write into memory. The full frames still exist in the episode log on disk, which is exactly what the feed below is rendering; they just never re-enter the prompt.

Watch him see

Below is the matrix from the live server — the most recent turn Emil took, drawn as an image by coloring each cell by block kind, darker meaning farther away. Straight up is the top edge, straight down the bottom, north the left edge. Hover or tap a cell to read the exact token the model read, or flip the control above the frame to the raw view to see the matrix exactly as it goes into the prompt, legend and all. The feed deliberately lags the world by about ten minutes and refreshes on its own while he plays; the full transcripts, this frame included, are at /emil-logs.

Forty percent of the prompt on a single sense is a lot, and I’m deliberately not trimming the frame itself by taste: a look that works at 14k tokens beats a cheaper one that points him the wrong way, and resolution can come down later, along a measured accuracy-versus-tokens curve. For now the matrix stands as it is — the world snapshot itself, cut down to what a pair of eyes standing at that spot could honestly resolve, in a grammar where seeing something and being able to act on it are the same token.