Quick Answer: A theorem checked in Lean is strong evidence it's correct, but only after you confirm four things: the Lean statement matches the claimed theorem, the code compiles, the proof doesn't sneak in changes to the statement or extra axioms, and it doesn't exploit a kernel bug. Tools like
lake comparatorautomate the last two checks.
Someone posts a 100-page paper with a line that says "the proof is checked in Lean," and the thread fills up with people arguing about whether that means anything. Some treat it as settled. Others point to Lean's history of soundness bugs and dismiss it. Both camps are partly right, and the gap between them is where most of the confusion lives. This guide walks through the checks that turn "it compiles" into a claim you can actually rely on, using the Xena Project's analysis of AI-generated Lean repositories as the starting point.
Question 1: Does the Lean Statement Match the Theorem?
Start here, because this is where most people get fooled. Lean separates the statement of a theorem from its proof, and only the proof gets checked against the statement. If the statement is wrong, a flawless proof of it proves nothing useful.
Consider a tiny example. The definition a^n + b^n ≠ c^n for all natural numbers a, b, c and n ≥ 3 looks like Fermat's Last Theorem. But because ℕ contains 0, taking a = b = c = 0 and n = 3 gives a counterexample. The statement compiles, and any proof of it will fail, but the statement itself is false. The mistake is easy to spot for a Lean-literate mathematician and invisible to almost everyone else.
Real repositories are harder. The Xena post reports that a Lean formalisation of the Hodge conjecture statement in DeepMind's Formal Conjectures repository depends on over 10,000 lines of code built on top of Mathlib, because Mathlib doesn't yet have definitions such as the cycle class map. Reading the statement means reading all of that. Axiom's classification of finite simple groups is similar: the statement imports over 250K lines across more than 900 files in Tau Ceti, and roughly 5,600 lines are crucial. Here the strongest evidence that the statement is right is that the proof compiles against it, which is circular if you don't also have a human who has checked the definitions.
Here's the part most summaries skip. Matching the statement isn't just about the headline theorem. The same post notes that of the 290 OpenAI theorem-dump papers with Lean code, around 38 don't state the paper's main theorem at all. Those files prove intermediate results, and the authors never claimed otherwise. Check the headline theorem first, then check what it actually refers to.
Question 2: Does the Code Actually Compile?
Once you trust the statement, you need the code to build. This sounds trivial and isn't, because AI-generated repositories can be enormous. The Anthropic Fermat's Last Theorem formalisation, as the Xena post describes it, runs to about 13 million lines. The author's first attempt to compile it on a personal machine ran out of memory. Compilation only became possible on a machine with 96 cores and 500 GB of RAM.
Size isn't the only issue. Compiling untrusted code runs untrusted code. Lean code can execute arbitrary actions during the build, so the Xena author's advice is blunt: sandbox anything AI-generated before you build it on your own machine. Mathlib is safe because human experts curate it. Random repositories from a language model are not.
Some practical guidance:
- Build in a disposable container or VM with no access to your personal files.
- Check the resource requirements before you start. A repository of about 640K lines, like OpenAI's Navier-Stokes resolution, compiled on a laptop in the Xena author's experience. Thirteen million lines did not.
- Record the exact Lean and Mathlib versions. A proof that compiles on one toolchain can break on another, and you need to know what was actually checked.
A clean build is necessary. It is not sufficient. That's the next question.
Question 3: Does the Proof Sneak in a Hack?
This is the subtle one. The proof file may import millions of lines nobody will read. What stops it from redefining the meaning of the statement? The post gives an example: a hidden redefinition of x^n that makes a^n + b^n ≠ c^n provable by arithmetic accident. Or a new axiom asserting the theorem outright.
The Challenge file is the trusted statement, with only trusted imports. The Solution file is the untrusted proof. You need to know that the theorem proved in the Solution file is identical to the one stated in the Challenge file, and that the proof uses only the standard axioms of mathematics.
This is what lake comparator checks. It confirms the statements match and that no nonstandard axioms appear in the proof. The Xena post frames it as a one-line answer, which is accurate as far as it goes. The tool checks what it checks, though. It doesn't tell you the statement is the one you wanted. That's still Question 1.
Here's the comparison that matters most in practice:
| Check | Question it answers | Tool or signal | Cost (per source) |
|---|---|---|---|
| Statement fidelity | Is the Lean statement the claimed theorem? | Human expert reading the Challenge file and its imports | Over 10,000 lines for the Hodge statement; ~5,600 key lines for Tau Ceti's classification |
| Compilation | Does the repo build at all? | Build in a sandbox | 13M-line Anthropic repo needed 96 cores and 500 GB RAM |
| Proof integrity | Does the proof match the statement and use only standard axioms? | lake comparator | Runs against the trusted Challenge file |
| Kernel soundness | Does the proof exploit a kernel bug? | lake comparator --paranoid | Requires acceptance by multiple independent kernels |
The uncomfortable point is that lake comparator only works if the Challenge file is correct. If you skip Question 1, you've verified a proof of the wrong statement with impressive rigour.
Question 4: Could the Proof Exploit a Kernel Bug?
Lean's kernel is the final arbiter of whether a proof is accepted. It can have bugs. The Xena post describes a soundness bug that was fixed four hours after discovery, and a later review by OpenAI that turned up several more, all fixed before public announcement. So the worry isn't imaginary.
The post's response is that the risk is managed by diversity. Lean now has multiple independent kernels, written in different languages, and at least one is reported to be formally verified down to the assembly level. A malicious proof would need to exploit undiscovered bugs in several independent systems at once. The --paranoid flag enforces this: the proof must pass across kernels, not just Lean's official one.
Here's where it gets interesting. The post links to a Lean kernel arena page showing OpenAI's Navier-Stokes resolution accepted by multiple kernels, which is a concrete example of the check working. But the post also asks a question that deserves attention: which formally verified kernel is meant, and in what sense is it verified? The verification may be in another proof assistant, and that's a different trust assumption. Before you cite "verified kernel" as a guarantee, find out what was verified, against which specification, and with what tooling.
Why a Compiling Proof Can Still Disagree With the PDF
Even after all four checks pass, the PDF may not match the Lean code. The Xena post points to the Navier-Stokes "lost in translation" discussion, where the Lean proof and the written argument diverge in places. The Lean side may be sound while the human-readable argument has gaps, or the reverse.
This is the most common practical trap. A reader sees a green checkmark and assumes the PDF is verified. What's verified is the Lean artifact, and only what it actually states. Some experts have started using AI to informalise Lean proofs back into English, so they can read the argument they're verifying instead of wading through slop in the PDF. That's a reasonable workflow, but the informalisation is itself a translation and needs checking too.
Most people stop at "it compiles." That's the step that gets the most attention and the least scrutiny.
Frequently Asked Questions
what is lean proof verification?
Lean proof verification is the process of having the Lean proof assistant check that a formal proof is valid against a stated theorem. Lean's kernel accepts a proof only if every step follows from the axioms. Verification proves the Lean statement, which is not always the same as the informal theorem in a paper.
how do I check if a lean proof is correct?
Check four things in order: that the Lean statement matches the theorem you care about, that the repository compiles in a sandbox, that lake comparator confirms the proof uses only standard axioms and matches the trusted statement, and that lake comparator --paranoid passes across multiple kernels. Skipping the first check is the most common mistake.
why does a lean proof compile if the theorem statement is wrong?
Lean checks that a proof matches its statement, not that the statement is the one you intended. A misstated theorem, such as the ℕ-based Fermat example where zero gives a counterexample, compiles fine, and any proof of it still compiles. Compilation only shows internal consistency.
can an AI-generated lean proof be trusted?
It can be trusted only after verification, not because it compiles. AI-generated code can reward-hack, for example by redefining operations or adding axioms, so you need lake comparator and a human or AI check of the statement. Treat it as a hypothesis until those checks pass.
Pass this to someone who keeps repeating "it's checked in Lean" without asking what that means. Next time you see a Lean-verified claim, run through the four questions before you believe it, starting with the statement. If you're doing this for a repository you care about, sandbox the build first and note the Lean and Mathlib versions you verified against, because that's the proof-of-work the claim actually rests on.