Iron Pickaxe Crafted

At four in the morning today, an autonomous agent running on the machine next to my desk crafted an iron pickaxe.

If you play Minecraft, that’s fifteen minutes of work. If you don’t: it doesn’t matter. This isn’t really a post about Minecraft — the game is just the lab bench. The agent is Emil, a small local LLM that lives on a survival server I host on a DGX Spark, with the long-term goal of living a full autonomous life there. And for two months, that pickaxe was unreachable. Not because the model didn’t know how to make one — every LLM has read the entire Minecraft wiki — but because of how I was engineering the system around it.

The pickaxe finally happened after I took two deliberate steps back. That’s the story.

---

We started fast, and trained on garbage

The obvious plan when you have a GPU and an agent: collect episodes, fine-tune, repeat. We started training LoRA adapters within the first days. It felt like progress. It wasn’t.

The early episodes were garbage in ways that only became visible later. The agent kept dying in the first minutes, so that’s what the data was — I eventually noticed we were training on behavior “applicable to 1% of the playtime.” Worse, world-specific facts were leaking into the weights: coordinates of a procedurally generated world, memorized. And the low point: a series of checkpoints fine-tuned on episodes with the reasoning stripped out. The result behaved exactly like what it was trained on — action patterns without decisions. It stood in the game staring into the void. We called it the lobotomy and threw three checkpoint versions away.

Lesson one, learned expensively: training amplifies whatever system produced the data. If the system is flawed, you are burning compute to make the flaws permanent.

Swapping models doesn’t fix your system

The next reflex was also the obvious one: maybe the model is the problem. We went through a small carousel of open-weight bases and landed on the best available at the time for this box, a fast Qwen 3.6 MoE.

The useful part of that phase wasn’t the model we picked. It was the rule that came out of it: serve the untuned base model for as long as possible, and put all the effort into the harness. A base model is a fixed, well-documented quantity. Every fine-tune is a fork of unknown behavior; every harness change invalidates the data you collected before it. You cannot debug both at once. Nothing is worse than a moving harness under a model you’re trying to train — you end up training the model to compensate for bugs you were about to fix.

So the weights froze, and the harness became the product.

The harness is the curriculum

Refining the harness sounds like plumbing. It turned out to be where all the actual insight lived.

The formative example: at one point ~90% of the agent’s tool calls were “errors,” and every intuition says the model is bad. The agent dug through the logs and found the opposite — there was an invisible movement budget in the harness, and the model was behaving rationally against a constraint it couldn’t see. The errors were the harness’s fault. Since then every “the model is dumb” episode gets the same treatment: assume the model is responding reasonably to what it can actually observe, then go find what it observed.

Out of dozens of cases like that came the design principle the whole codebase now follows: the harness states facts, it never teaches the game. No hints, no tutorials in error messages, no helpfully widening a parser because the model formatted something wrong. The model arrives knowing Minecraft; knowledge is its job. The harness’s job is to be a truthful, minimal interface — crystal clear, nothing else. Every time we “helped” in the harness, we created a behavior we could never train away later, because the data would carry the crutch.

Two disciplines made this survivable:

Every LLM interaction is captured in evals. Each observation the model sees and each tool result it gets back is a surface, and every surface change runs against a replay suite — hundreds of real captured cases — plus a scorecard over live episodes. Before that suite existed, 40% of failing calls were retried verbatim and we argued from anecdotes. After, a regression was a number.

Scenarios live in isolated dojo worlds. Instead of harvesting the chaos of live play for training data, we mapped the majority of situations — fight this mob, survive this night, craft toward this goal — into isolated, repeatable drill worlds with world-truth success predicates. The reframe that unlocked it was refusing my own pipeline: why collect data from a flawed system with a dozen uncontrolled factors? The dojo now holds 68 staged scenarios that can manufacture clean trajectories at hundreds of rollouts per hour, ready for the day training resumes. The live server is for living; the dojo is for learning.

The expensive lesson: autonomous refinement

Here’s the part I’m least proud of, and the most useful to share.

For a while I ran autonomous sessions where a frontier-model agent watched Emil live, with standing permission to tweak the harness whenever he got stuck. It sounds like the future: a stronger model continuously unsticking a weaker one. What it actually produced was drift. Every wrong tool call became a reason to move the harness — a new config knob, a special case, an “oracle” channel quietly feeding the model answers. Each individual change looked defensible. The sum was an overengineered system nobody had designed, and the constant change through babysitting was itself destroying the agent’s behavior — you can’t act coherently on an interface that shifts under you.

That was poor engineering, and it was mine: I had delegated judgment — what good behavior looks like — to a system that only had judgment about code. It cost me days of personally reading every line to unwind. The cleanup deleted twelve config knobs and every oracle channel in one 129-file cull, and it left a standing rule: agents bring diagnosis, options, and data; every harness change goes through me. Autonomous agents are spectacular legwork — log forensics, measurement, implementation. They are not the ones who get to decide what the system should be.

Two steps back

So the state going into this weekend: no fine-tuning (step back one), no harness magic (step back two). A frozen, radically simplified interface, an eval suite guarding every surface, a dojo full of unused training scenarios, and untuned base weights.

Then we swapped the base model one more time — to a newer, dense 27B that runs six times slower than its predecessor on this hardware. On paper, a downgrade. Its first live session was a mess of endless reasoning loops, and the fix was almost insulting in its smallness: we were running the sampling profile of the previous model. Setting the temperature and reasoning effort the model card asks for — one evening of diagnosis, a handful of lines — took failing calls from 90% to 10%.

That night, on stock weights, over a single unbroken 23-hour episode, Emil died 23 times in one bad hour to a mob camping his respawn point — then rebuilt from nothing, worked out the full tool chain, wrote the milestone into his own diary, and went ten and a half hours without dying. The iron pickaxe was crafted at 04:00. In two months of trained checkpoints and clever harness interventions, he had never gotten past stone.

What I’d tell past me

• Don’t train until the system generating your data is one you’d freeze. You’ll know it’s ready when you’ve stopped wanting to change it daily.
• The harness is not scaffolding around the real work. It is the real work — it’s the model’s entire reality, and every distortion in it becomes a distortion in behavior, then in data, then in weights.
• Put an eval behind every surface the model touches, before you think you need it.
• Let agents do the forensics and the implementation. Keep the judgment. The moment an autonomous loop is editing the environment it’s evaluated in, you’ve built a feedback system with no damping.
• Progress can look like removal. Almost everything that finally worked was something we deleted, simplified, or refused to do.

The model that crafted the pickaxe has never been trained by me. That’s the point. Now that the system around it holds still, training finally has something worth learning from.

The iron pickaxe sprite is a Mojang asset from Minecraft, used here in fan content. Minecraft is © Mojang/Microsoft; this blog is not affiliated with or endorsed by them.