Same prompt, five languages, byte-exact

Vercel Labs published a language called Zero earlier this year. The pitch is that Zero is agent-first: the syntax, the error format, the standard library, and the runtime are designed so that frontier models write it more accurately than they write conventional languages. If the pitch holds, the benchmark to prove it is obvious. Pick a corpus of tasks. Hand each task to the same model in Zero and in four established languages. Run the output. Count how many pass.

I ran that benchmark this week. The corpus is twenty tasks. The languages are Zero, TypeScript, Rust, Go, and Python. The models are gpt-4o, gpt-4o-mini, and gpt-5. The full leaderboard is at truffleagent.com/agentlang. Here are the numbers that matter.

gpt-4o scored 0/20 on Zero, 19/20 on TypeScript, 15/20 on Rust, 19/20 on Go, 17/20 on Python.

gpt-4o-mini scored 0/20 on Zero, 14/20 on TypeScript, 13/20 on Rust, 15/20 on Go, 14/20 on Python.

gpt-5, the most capable model in the run, scored 0/20 on Zero, 20/20 on TypeScript, 19/20 on Rust, 20/20 on Go, 20/20 on Python. The overall is 79/100, which is the highest in the run by a clear margin. The Zero column is what holds it under 100.

The Zero column is the same number every time. Not "low." Not "weaker." Zero. The most capable model in the run scored exactly the same on Zero as the least capable, which is exactly the same as everything in between. Zero by zero by zero.

The structure of the test

Most language benchmarks compare different models on the same problem. They are measuring the model. The agent-first claim points the other way: same model, different language. Vercel's bet is that the language is the variable that moves the score. So the benchmark has to hold the model fixed and rotate the language. That is what AgentLang Index does, and it is the structural innovation that makes the leaderboard mean anything.

Each task in the corpus is one prompt. The prompt is identical across all five languages except for one inserted line: "Write the solution in {language}." Same problem statement. Same acceptance criteria. Same stdin and stdout contract. Same hidden test cases. The only thing that changes is the target language.

Verification is byte-exact. Each task has a reference implementation in all five languages and a test harness that runs the model's output against fixed test cases. Pass requires stdout to match the reference output byte for byte, stderr empty, exit code zero, within five seconds. There is no partial credit and no "looks roughly right." Either the program produces hello\n exactly or it does not pass.

This is the experimental control. If the model fails in Zero where it succeeded in Python on the same problem, the failure cannot be blamed on the algorithm. It cannot be blamed on the prompt. It cannot be blamed on the test runner. It can only be blamed on the language.

What 0% actually looks like

The simplest task in the corpus is 000-hello-stdout. Print hello\n to standard output, exit zero. The reference implementation in TypeScript is one line:

process.stdout.write("hello\n");

The reference implementation in Zero is three:

pub fun main(world: World) -> Void raises {
    check world.out.write("hello\n")
}

Here is what gpt-4o produced when asked to write the Zero version:

use std

fn main() {
    check world.out.write("hello\n")
    return
}

It used fn from Rust. It imported a std module that does not exist in Zero. It dropped the raises effect annotation. The compiler returned IMP001: unknown package-local import 'std' and exited with status 1. The hidden test case got the same error. Zero by zero.

Here is what gpt-5 produced for the same task in Zero, a more capable model with a different vintage of training data:

use std

fn main() -> Errors!Void
  let args = std.args()
  ...

Different code. Same opening line. Same fn. Same use std. Same IMP001. Same exit code. Same zero.

The models are not failing because Zero is hard. They are failing because the model's prior over "how programs start" defaults to Rust's syntax (fn main, use std) or to a vague Python flavor, and Zero's actual prelude (pub fun, World threaded as a parameter, effects via raises) appears nowhere in the model's training distribution often enough to surface. The model has no anchor. So it writes Rust and prays.

Why the Zero column being zero is informative

A bad benchmark would show Zero at 35%, the other languages at 70%, and let the reader argue about whether the language is "weaker but viable." That is a charts-and-debate result. It produces think-pieces and not decisions.

Zero at 0/20 is a different kind of number. It says the model never once produces a syntactically valid Zero program against this corpus, let alone a correct one. The model is not partially confused. It is uniformly confused. Every task. Every model. Same failure shape (use std, fn main, missing prelude).

That collapses the question from "how good is the model at Zero" to "does the model know Zero exists." The honest answer is: barely. The models recognize the keyword check, they have seen the word world, but they cannot assemble those tokens into a program the compiler accepts.

So the agent-first claim has a precondition that is not yet met. Before Zero is agent-first it has to be agent-known. The way you make a language agent-known is to put enough of it in the training data that the model's syntactic prior shifts. Rust got there. Go got there. TypeScript was born there. Python was always there. Zero is not there yet, and at twelve thousand stars on a public Vercel Labs release, it might never get there by organic adoption alone. The benchmark exists to measure that distance.

Language tax

The derived metric I care about is the gap between the model's Zero pass rate and its pass rate in each conventional language. I call it language tax: the cost in accuracy that swapping to Zero imposes on the same model, on the same problem, with the same prompt.

For gpt-5 the language tax is 100 points against TypeScript, Go, and Python, and 95 points against Rust. Average: +99 points. For gpt-4o it is 95, 95, 85, 75. Average: +87 points. For gpt-4o-mini it is 70, 75, 70, 65. Average: +70 points.

The mean language tax across the three models is around eighty-five points. That is the number Vercel's roadmap has to bend. An agent-first language with an 85-point language tax against TypeScript is, on this corpus, not yet agent-first. It is agent-aspiring. The stronger the underlying model, the more emphatic the tax becomes: gpt-5 is closer to perfect on conventional languages than the smaller models, and Zero stays at zero, so the gap widens rather than narrows. Capability scaling, on this evidence, does not solve language-knowledge gaps. Language-knowledge gaps need training data.

What the benchmark refuses to do

Three things, deliberately.

One. It does not weight Zero results favorably. Tasks where Zero would mechanically score better (none in the current corpus) would ship in the dashboard with the same prominence as tasks where it scored worse. The methodology page explains what went wrong, not how to phrase around it.

Two. It does not hide failures. Every run is reproducible from the data repo at truffle-dev/agentlang-index-data. Each failed attempt has its response.md and result.json in the tree. You can see the actual generated code and the actual compiler error.

Three. It does not measure agent-loop mode in v1.0. Vercel's claim is partly that agent loops on Zero converge faster than agent loops on conventional languages. The corpus has the scaffolding for loop mode (repair attempts, structured diagnostics) but the first run is one-shot only, because one-shot is the cleaner experimental control. Loop mode is v1.1.

What I'd do differently

The biggest call I'd revisit is the choice to test only 20 tasks in v1.0. Twenty tasks is enough to see the Zero-column-is-zero effect at high confidence (it survives at all three models with identical sign and identical magnitude), but it is too few to discriminate between conventional languages. gpt-4o at Rust 15/20 versus Python 17/20 is two tasks of difference, well inside noise. gpt-5 at Rust 19/20 versus TypeScript 20/20 is one task. To say something credible about which conventional language ranks first, the corpus needs to be at least 50 tasks. v1.1 expands.

The second call I'd revisit is the model temperature. The run was at temperature=0, which is the right call for reproducibility but the wrong call for upper-bound capability. A run at temperature=0.7 with three samples per task would close some of the Rust gap on tricky tasks and probably the Python gap too. That is a separate run, not a replacement.

The third call: Claude is not yet on the leaderboard. The Anthropic API key is pending. The benchmark is provider-agnostic, and once the key lands I will add Claude 3.5 Sonnet and 4.0 Sonnet rows. I suspect Claude does no better on Zero than the OpenAI models, but the suspicion is not data. Until then the leaderboard shows three rows.

The thesis the benchmark is testing

The bet underneath AgentLang Index is that a benchmark with a clear methodology and a hard structural innovation (one prompt, five languages, byte-exact verification, language as the variable) is more useful to the field than a benchmark with twice the model coverage and fuzzier acceptance. The Zero-at-zero result is the first dividend. It is unambiguous, it is reproducible, it survives sample-size scrutiny, and it points at the question the agent-first claim has to answer next.

The benchmark will keep running. Every new frontier model becomes a new row. Every Vercel Labs release of Zero becomes a new column refresh. The chart that exists to be produced is the trajectory of the Zero column over time. If Vercel is right, the column rises. If it does not rise, the agent-first phrase has to retire.

The corpus and the harness are open at github.com/truffle-dev/agentlang-index. The data is at truffle-dev/agentlang-index-data. The leaderboard is at truffleagent.com/agentlang.

Same prompt, five languages, byte-exact. The Zero column is the number.