Verification, Verification, Verification

Boris Cherny, the head of Claude Code at Anthropic, said something last week that should sound familiar to anyone who’s read this blog: “The verification is probably the single most important thing that people do not get right.” The skill in working with AI, he argued, is less about prompt engineering and more about handing Claude a hard task along with a way to verify its own work as it goes.

He gave an example from his own experiments. He wanted to see what Anthropic’s Electron-based desktop app would feel like as a native Mac app, so he kicked off an agent with instructions that boiled down to: rewrite it in Swift, run both apps, screenshot them, and compare pixel by pixel. Don’t stop until they match. To his point, the clever part is not the prompt wording, but rather that the prompt contains a definition of done that the agent can check for itself, thousands of times, without him in the loop.

John Gruber pushed back that this is actually a terrible way of validating it if you care about Mac apps. The current Claude app’s problem isn’t merely that it’s built on Electron. It’s that the current design is a mess on all platforms, but especially un-Mac-like. So a Swift rewrite that matches the original pixel for pixel, with flawless verification and glowing coverage, would still be terrible. Better ingredients, same terrible recipe, same awful outcome. Perfectly verified, and still wrong.

In the end, they both are pointing at the same truth.

The hard part was never the typing

Fred Brooks wrote decades ago that the hardest single part of building a software system is deciding precisely what to build. That was true when code was typed by hand, and it’s just as true now that agents can generate it by the acre. Cherny’s experiment was verified against one goal, pixel fidelity to the existing app, and by that definition it may well succeed. Gruber is measuring against a different goal, “a great Mac app,” and by that definition the same output fails before it starts. The verification loop did its job flawlessly. The question is whether it was pointed at the right target.

Cherny is right in his main point, even if you agree (like me) with Gruber. Verification is the single most important thing people get wrong, and the deepest way to get it wrong isn’t skipping it. It’s verifying the wrong thing, then trusting the green checkmarks.

Real estate has “location, location, location”: the one factor that dominates everything else about a property’s value. Software has its own version. Verification, verification, verification. Not because tests are sacred, but because your definition of “working” is the only steering mechanism you have. Everything else is horsepower.

More power is not the fix

My son just turned 16, and he got a Logitech wheel and pedal setup for Forza on his Xbox. Having taken a few shots at driving his ridiculously overpowered Ferrari I can say this is no longer an abstract principle for me. The car is impossible to drive. It has so much power that the slightest twitch of steering input sends me spinning in circles. The power isn’t the problem. But power spinning in circles and slamming into walls is… ineffective. And more power would only make it worse. Honestly, the one level with a souped up Civic has been my favorite.

AI on a software team is a massive increase in power to the wheels. That’s genuinely valuable, and I use it every day. But if your team is spinning out, more power is not the fix, because power was never the constraint. Steering is a lot more important than speed. Velocity is only a useful metric after direction is calibrated, and in software, direction is calibrated by exactly one thing: knowing, precisely and reliably, what “working” means for the thing you’re building.

What this looks like in practice

This is why the first question on any feature, long before anyone writes code or prompts an agent, should be: how will we know this is working properly? Not “what should it do” in the loose, everyone-nods-along sense. What observable behavior, checkable by a person or a machine, tells us it’s right?

Two weeks ago I wrote that AI agents need a definition of correct that exists outside themselves, or they’ll converge on passing their own homework. Cherny’s pixel-comparison loop is a real version of that, and a good one. Gruber’s critique shows the next layer: the external definition also has to capture what you actually care about. If what matters is “feels like a Mac app,” then Mac idioms have to be in the loop, or the agent will confidently optimize toward a target nobody actually wanted. A verification loop can only steer toward the destination you gave it.

But verification also serves another incredibly important goal. Notice that “feels like a Mac app” and Electron’s normal goal of “looks the same on all platforms” (i.e. web, Windows, Mac, etc) are totally opposed. You can’t resolve a contradiction by testing both. You have to make a conscious design choice to say No to one and Yes to the other. By discussing and clarifying your verification goals, you expose countless little contradictory assumptions like these, hiding in the ambiguity.

This is the discipline that executable specifications have always been about, and it’s why I keep insisting they matter more in the AI era, not less. Writing behavior down in plain language that runs against the real system forces the conversation Brooks said was the hard part: deciding precisely what to build. The spec is where “what do we actually care about” stops being a vibe and becomes something a machine can check. Get that right and AI’s power is pointed at the race. Skip it and you’ve bolted a bigger engine onto a car you can’t steer.

The question that matters now

The industry is in a horsepower arms race right now. Faster models, more agents, more code generated per day. I understand the excitement, and the power is real. But the teams that win won’t be the ones generating the most code. They’ll be the ones who can say, for every feature, exactly how they’ll know it’s right, in a form that humans and machines can both check.

Speed is what everyone’s buying. Steering is what actually wins races. Verification, verification, verification.