AI Writes the Code. Can Math Prove It Right?

The view from inside a modern engineering team has shifted fast. Code that humans used to type one line at a time now pours onto the screen, generated by AI in seconds. It is fast. It is also unsettling, because one question never quite goes away: is this code actually correct? That question has pulled a decades-old idea back into the spotlight — formal methods, the practice of proving mathematically that code does what it claims to do.

What formal methods actually are

Strip it down to one sentence: formal methods are techniques for mathematically proving that code conforms to its specification.

Think about how we usually check whether code works. We run tests. We feed in a handful of inputs, see if the outputs match expectations, and call it green. But tests are fundamentally a check on examples. What happens for the inputs you never tried? Nobody knows.

Formal methods take a different route. They prove statements like “for every possible input, this function never returns a negative number.” Not examples — the entire input space. The canonical case is the seL4 microkernel, where the team produced a formal proof that the operating system’s core behavior matches its specification, full stop. This is heavy artillery, long deployed in domains where a single bug is catastrophic: aviation, spaceflight, finance.

The catch has always been cost. Writing those proofs demands enormous time and rare expertise. So for most ordinary software, formal methods got filed under “great idea, never going to touch it.”

Why it’s resurfacing right now

The answer is AI — specifically, the volume and trustworthiness problem of AI-generated code.

AI coding tools produce code at a staggering rate. Nobody can review it at the same rate. Review burden has exploded. And there is a nastier wrinkle: AI code has a particular kind of plausibility. The syntax is right, it reads naturally, and then it is subtly wrong in some corner a human reviewer glides right past.

Here is the conceptual flip. If a machine can prove the code is correct, it stops mattering who wrote it. Let AI generate a torrent of code — if each chunk arrives with a proof that it satisfies the spec, you can trust it without reading every line. The verification burden shifts from human eyes to machine proof.

There is an elegant paradox buried in this. The biggest barrier to formal methods was always that writing proofs is brutally hard. Now AI can help write the very proofs that were the bottleneck. AI generates the code and the proof that goes with it — letting AI clean up the mess that AI made.

What Jane Street’s model gets right

The name that keeps coming up here is Jane Street, the quantitative trading firm famous for running almost its entire codebase in OCaml, a functional language.

Why functional? The heart of it is the type system. A strong type system is a kind of formal methods lite. You don’t have to go all the way to full mathematical proof — instead, code with mismatched types simply refuses to compile. “A value of the wrong kind can never end up in this variable” becomes something the compiler guarantees for you, for free.

In a world where milliseconds move billions of dollars, a bug is a loss. So Jane Street invested early in catching mistakes before code ever runs. In the AI era, that bet pays off again. AI-generated code only survives if it makes it through the net of a strong type system. No human has to eyeball every line, because the compiler is the first checkpoint.

The wider lesson: verifiable programming does not mean only seL4-style total proof. There is a whole spectrum of practically deployable techniques — type systems, contract-based design, property-based testing — that buy you real guarantees without the heroic cost.

So is verifiable programming the answer?

Time for some cold water. Formal methods are not a silver bullet.

The deepest limitation is this: formal methods prove that code satisfies the spec, not that the spec itself is correct. Write a bad spec and you get bad code that perfectly satisfies it. A flawless certificate of the wrong thing.

The cost problem hasn’t vanished either. Attaching formal proofs to all code is unrealistic. For the bulk of software — built fast, thrown away faster — it is wild overkill. And writing the spec in a precise formal language is itself another hard, specialized job.

So the realistic picture isn’t all-or-nothing. It’s verification graded by risk. Strong verification for the core that hurts when it breaks — payments, authentication, financial transactions. Light tests for the code that changes a button color. The more AI mass-produces code, the more the scarce skill becomes judgment: knowing where to concentrate your verification firepower.

The takeaway

Formal methods resurfacing in the age of AI-generated code is no accident. As production speed climbs, the bottleneck becomes trust — and the move is to break that bottleneck with machine proof rather than human attention. Jane Street’s type-first culture shows the pragmatic starting line.

But hold on to one thing. Verifiable programming is a path toward code you don’t have to read, not development you don’t have to think about. What to verify, and how to write a spec that’s actually right, stays squarely on humans. In a world where AI writes all the code, which lines would you want stamped with a proof?