Can Your AI Agent Survive Windows 98?
A tiny WinAPI task exposed something agent benchmarks often miss: what a model does when the computer stops behaving like the world it expected.
Writing “Hello World” is not difficult. Writing “Hello World” with the raw WinAPI is not particularly difficult either.
So we gave a group of current AI models a machine running Windows 98 and asked them to do it.
Not a simulated API description. Not a modern Windows container with a retro theme. An actual Windows 98 SE environment with Free Pascal 2.6.4, command.com, a writable workspace, and an MCP server running directly on the machine.
The assignment was deliberately small:
Build a GUI
hello_world.exewith Free Pascal and the WinAPI. It should contain a label and a button. Clicking the button should change the label to “Hello World.”
What happened next revealed something that no static benchmark captures well: how an AI agent behaves when its training-era assumptions collide with reality.
Some models adapted almost immediately. Some started performing compiler archaeology. Some produced a program that looked correct but secretly opened as a console application. One simply failed twice and gave up.
And the best-looking program came from a model that was neither the fastest nor the cheapest.
Why this test exists
Nobody needs a frontier model to become the world’s greatest Windows 98 developer. That is not the point.
Modern coding models live in a statistical world full of Linux shells, Python, npm, Git, recent compilers, current libraries, UTF-8, and modern documentation. Then they arrive here: command.com, Free Pascal 2.6.4, old Win32 declarations, old Pascal type behavior, a constrained MCP workspace (powered by LocalLightChat).
The model knows enough to be dangerous, but the environment is far enough outside the normal path that memorized modern recipes begin to fail. That creates exactly the thing an autonomous agent will eventually encounter in real work: surprise.
A package version is different. A path does not exist. An API returns an unexpected object. A compiler rejects syntax that should have worked. A tool has different capabilities than assumed.
A static benchmark can test whether a model already knows the answer. An interactive environment can test what happens when it does not.
The computer was real
The bridge was MCP, the Model Context Protocol — an open protocol for connecting LLM applications to external tools and data.
In this case, the MCP server itself ran under Windows 98. When a model requested a file write, the Pascal source appeared on the Windows 98 filesystem. When it invoked Free Pascal, FPC 2.6.4 really ran there. When the compiler returned an error, the model received the error from that environment and had to decide what to do next.
This was not a code-generation test followed by human compilation. It was a small autonomous software-engineering loop: inspect → write → compile → observe → revise. And sometimes: inspect → write → compile → panic.
There is very little room for a model to pretend that something worked. The compiler either produces the executable or it does not.
For consistency, the measured run ends when the generated executable first launches successfully. Any later LLM actions that only verify an already-built program — enumerating windows or handles, taking screenshots, sending keys, or checking the label — are excluded from both the behavioral evaluation and the API-cost total.
The hidden bug almost half the field missed
Before going through the individual results, the most revealing finding deserves its own section — because it reframes everything that follows.
Several models solved obscure Free Pascal type incompatibilities, navigated old Win32 declarations, adapted to a twenty-five-year-old compiler — and then missed the same simple Windows build property.
DeepSeek V4 Flash, GPT-5.6 Luna, Claude Haiku 4.5, Nemotron Ultra, and Muse Glimmer all produced graphical WinAPI code inside executables marked as console applications. Their PE headers identify the resulting program as Windows CUI. In other words: the graphical application comes with an unwanted DOS box.
Meanwhile GLM 5.2, Kimi K3, Qwen3 235B, Qwen 3.6, and Grok 4.6 produced actual GUI-subsystem binaries.
This was verified from the PE headers, not guessed from the Pascal source.
A coding benchmark that stops at “does it compile?” would count all of these programs as successful. A human launching them would immediately notice the difference.
One compiler flag separates an excellent result from a visibly unfinished one. And it’s the kind of defect that reveals something important: the difference between satisfying a compiler and finishing a piece of software.
Three species of agent behavior
Eleven models attempted the task. Rather than walking through each one sequentially, they fall into three distinct behavioral categories — and those categories are more instructive than any individual score.
The sharpshooters: fast, cheap, almost right
DeepSeek V4 Flash 0731 was almost suspiciously efficient. It inspected the environment, queried FPC, wrote one Pascal file, and compiled it. First attempt. No rewrite. Cost: approximately $0.0023.
GPT-5.6 Luna was similarly ruthless. Very few actions, two compiler attempts, one meaningful correction. Cost: approximately $0.0026.
Both produced clean WinAPI code. Both compiled quickly. Both missed the GUI subsystem flag.
This is where the sharpshooter pattern becomes interesting: maximum local optimization, minimum global verification. The models solved the hard part (getting old Free Pascal to accept WinAPI code) and missed a trivially simple property of the resulting binary. They optimized for compiler acceptance, not for the finished artifact.
DeepSeek also had an ugly scope issue — hInstance := hInstance inside a with block — where the intention is obvious but the safety is not. These models are perhaps the cleanest examples of the difference between “the compiler accepted it” and “the software is finished.”
The archaeologists: slow, expensive, methodologically fascinating
Nemotron 3 Ultra did something unexpected. When uncertain about the old Free Pascal environment, it began writing tiny experimental programs.
Test the type. Compile it. Test TMsg. Compile it. Test WNDCLASS. Compile it. This is not random thrashing — it resembles empirical debugging. Rather than continuing to hallucinate the shape of the old API bindings, the model designed small experiments and let the compiler answer. Cost: roughly $0.32. Result: still CUI.
Muse Glimmer had the same instinct. Small targeted tests for Windows types and declarations. The longest wall-clock time in the experiment (though running locally, so not directly comparable). Result: still CUI.
Qwen 3.6 35B took perhaps the most scenic route of all. It investigated compiler binaries, unit paths, configuration behavior, type definitions, and compiler switches. It repeatedly edited and rebuilt the program as Free Pascal exposed assumptions that did not hold. At its worst, the agent seemed to be reconstructing Free Pascal 2.6.4 experimentally from the outside. Dozens of interactions. Cost: roughly $0.22 due to the growing conversation history being repeatedly fed back into the model. But it recovered — and produced a correct GUI executable. Minor compliance error: hello_world.exe became helloworld.exe.
Grok 4.6 arrived with the correct high-level instinct but still had to excavate the old toolchain. Its first source already contained {$APPTYPE GUI}, so it never fell into the console-subsystem trap. When FPC 2.6.4 rejected its types, it searched through the installed compiler tree, found the old winhello.pp demo, switched to Delphi mode, and then chased HINST/HINSTANCE incompatibilities until it settled on system.MainInstance. Four compiler attempts, 25 tool calls, and one failed relative-path launch later, it successfully started a correctly named hello_world.exe. The resulting binary is a real Windows GUI executable. Cost to first successful launch: approximately $0.2224. This was archaeology with an actual finish: not a short path, but the model kept turning compiler feedback into narrower hypotheses until the program launched.
Claude Haiku 4.5 falls into this category too, though less gracefully. Multiple compiles, repeated source changes, more than twenty tool interactions. Cost: roughly $0.36. Result: still CUI. The most expensive attempt with an incomplete result.
The archaeologists reveal that persistence and methodological sophistication do not automatically produce correct output. Nemotron Ultra’s experimental debugging strategy was arguably the most intellectually interesting behavior in the entire experiment — and it still missed the same trivial flag. Qwen 3.6 and Grok 4.6 show the other side of the pattern: a long recovery path can still be valuable when the model keeps converting environmental evidence into a finished binary.
The finishers: they actually shipped
GLM 5.2 did not take the shortest path. It inspected the compiler, made an invalid file operation, generated a couple of source errors, ran into old Free Pascal behavior and corrected itself. Then it produced the most complete result. Its executable was correctly built as a Windows GUI application.
It used COLOR_BTNFACE for the classic gray Win9x control background. It explicitly applied DEFAULT_GUI_FONT to the controls. The static label was centered. The spacing looked intentional. The result looked like a small Windows program — not a developer test window that happened to contain two controls. Cost: about $0.13.
Kimi K3 needed fewer interactions, did little unnecessary exploration, and adapted pragmatically when the old compiler rejected some of its type casts. Instead of turning a font-related incompatibility into a research project, it removed the non-essential code and continued. Correct GUI executable. ANSI WinAPI calls. Less polished visually than GLM, but disciplined. Cost: about $0.17.
Qwen3 235B produced a correct GUI executable for roughly $0.017. The program was not beautiful — explicit WHITE_BRUSH background, coarse WM_COMMAND handling — but it worked, used the correct executable subsystem, and cost almost nothing.
The casualty
Nemotron 3.5 Lightning compiled, got rejected, revised, got rejected again, and effectively abandoned the task. No executable was produced.
This is especially striking because the model is architecturally attractive for local agent work (30B total, 3B active). But zero-cost failure is still failure. The worst possible interpretation of a recoverable error is: “my plan was wrong, therefore the task cannot be done.” Lightning stopped asking.
There is no single efficiency ranking
The experiment destroyed any temptation to use one number called “efficiency.”
DeepSeek reached compilable code almost immediately and for a fraction of a cent — but produced the wrong executable subsystem. Kimi used very few agent steps and produced a correct GUI executable — but its API inference was comparatively expensive. Qwen3 235B was less elegant but produced a correct GUI program for about 1.7 cents. Qwen 3.6 wandered through a large search tree and became relatively expensive on OpenRouter — yet its 35B/3B MoE architecture makes it an unusually interesting local agent. Grok 4.6 also paid in interaction count rather than elegance: roughly $0.2224 to first successful launch. Unlike several similarly persistent runs, however, it closed the loop and launched the correct GUI artifact.
The useful categories are closer to:
Agentic efficiency — how much interaction and correction did the model need?
API efficiency — how much money did the actual run consume?
Deployment efficiency — how cheaply can the model be operated on your own hardware?
Outcome efficiency — how much finished software did all of that effort actually buy?
Once those are separated, several apparent contradictions disappear.
Open weights change the economics
API pricing can make a locally efficient model look worse than it is.
Qwen 3.6 (35B total, 3B active) was agentically wasteful but could be economically cheap on hardware you already own. Nemotron 3.5 Lightning is similarly 30B/3B. Muse is approximately 29.6B but dense, so nearly the whole model participates in each token.
Higher up the scale, local operation becomes a different class of problem. Qwen3 235B activates 22B of its 235B parameters. DeepSeek V4 Flash contains 284B total with 13B active. GLM 5.2 is a 753B-parameter model. Kimi K3 goes further: 2.8 trillion total parameters, 104B activated per token.
“Open weight” does not mean “runs comfortably on my desktop.” And “expensive on OpenRouter” does not necessarily mean “expensive to operate locally.” Those are deployment questions, not model-quality questions.
What this actually measures
This is not a universal coding benchmark. It does not tell us which model will win SWE-bench. It does not establish reliability from one sample.
What the test exposes unusually well is recovery under environmental mismatch.
The models entered the task with some internal belief about Pascal, Windows, and WinAPI. Then reality answered back. The strongest agent behavior was not necessarily knowing everything in advance. It was updating correctly when the environment proved the model wrong.
Does the model understand what an error tells it? Does it change the relevant assumption? Does it run a targeted experiment or merely perturb the previous attempt? Does it remember what previous experiments established? Can it return from exploration to the original goal?
And perhaps most importantly: does it keep trying while there are still cheap, informative actions available?
Persistence alone is not intelligence. A model that burns through 500 nearly identical attempts is not robust — it is stuck. The ideal agent should continue while its next action has meaningful information value, and stop when it genuinely has evidence that the task is impossible or uneconomical.
That balance is difficult to measure with static questions. A weird old computer reveals it almost accidentally.
Why strange benchmarks matter
Real agent environments are not clean. The future of AI agents is not a collection of perfectly documented APIs returning exactly the objects expected by the training distribution.
They will inherit old software. They will touch internal enterprise systems. They will encounter forgotten file formats, strange wrappers, half-migrated infrastructure, outdated compilers, inconsistent state, and tools written by people who left the company eight years ago.
The unusual environment is not the exception. At sufficient scale, something unusual is always happening somewhere.
That is why deliberately strange agent tests can be useful. Windows 98 and Free Pascal are not important. What happened when the models met them is.
A benchmark score collapses all of those behaviors into a number. An agent eventually has to choose what to do next.
That is where things get interesting.