OpenAI's COBOL Migration Score Jumped From 5.4% to 86.6%. The Tests Are Hidden

• Book: AI Agents Pocket Guide
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

You hand a COBOL program to a model and ask for Java. The Java compiles. The tests you wrote pass. Then one batch run pays a customer one cent more than the mainframe would have, because the old code dropped the third decimal and the new code rounded it.

That cent is the bar OpenAI set in a post published on October 19 about coding in COBOL, Fortran and Visual Basic. In its COBOL migration eval, a migration only counts if every hidden test produces byte-for-byte the same observable output as the original. On that eval, OpenAI reports that GPT-6.1 Sol at low reasoning effort passed 86.6% of tasks, against 5.4% for GPT-5.6 Sol at the same setting (OpenAI, TokenPost). At high effort it reached 97.9%.

Those are large numbers from OpenAI's own evals, and the migration tests aren't published. Only one of the two evals is something you can rerun.

What OpenAI announced

The post is a capability update, not a new product. Runtime Wire describes it the same way, and OpenAI's own wording is that you "can try our latest models through the API on coding tasks in the languages your team uses."

It covers two evals:

• MultiLangBench (building). The model writes a SQLite-compatible database engine from scratch, in one of 20 languages (OpenAI).
• COBOL migration. The model moves a COBOL program to C++, Java, Kotlin, Python or Rust, and hidden tests check byte-for-byte parity on observable outputs (OpenAI).

OpenAI frames the work around languages that run critical systems, citing COBOL's share of banking, card and ATM transactions and Fortran's use in weather forecasting and nuclear safety analysis (OpenAI).

The numbers, model by model

On MultiLangBench at maximum reasoning effort, OpenAI reports these average test pass rates across the 20 languages (OpenAI; the GPT-5.6 Sol and Astra averages and COBOL cells also appear in TokenPost):

COBOL is Astra's lowest-scoring language, at 72.1% (OpenAI). Luna is the cheap option: OpenAI estimates $0.77 per sample against $3.48 for GPT-5.6 Sol, with much lower scores than Astra.

Price matters here because the two headline models sit far apart. GPT-6.1 Sol lists at $2 per million input tokens and $10 per million output. GPT-6 Astra lists at $10 and $50. For the migration eval, OpenAI says GPT-6.1 Sol ran at 61% of GPT-5.6 Sol's estimated API cost (OpenAI). TokenPost notes that this figure covers the eval only, not the cost of moving a production system.

One half of this you can run

MultiLangBench is on GitHub under Apache-2.0. Its README says the model must implement SQLite 3.40.1 in the requested language, with SQLite's documentation as the reference. The tests were converted from SQLite's TCL suite, about 24k of roughly 51k. Each test is an initial database, a user query and a validation query.

A few rules in the README are worth knowing before you compare your numbers with OpenAI's:

• Use only the 272 tests whose names start with 00, 01 or 02. The full suite is allowed for extra precision on high-scoring samples, as long as you say so.
• Strip every trace of SQLite from the environment first, and run with no internet access.
• Each test gets a 60-second timeout, and a solution in the wrong language scores 0.

The score is a pair, API cost and test pass rate, so a cheaper model with a lower score is not automatically worse for your budget (MultiLangBench).

The migration eval is the half you can't rerun. Runtime Wire calls them "OpenAI-designed hidden tests," and the GitHub repo covers only the SQLite build task. So the 86.6% is OpenAI's measurement against OpenAI's own hidden tests. OpenAI also says migration output still needs expert review and testing against business rules and system dependencies before it goes to production (OpenAI).

Build the parity check yourself

What you can reuse from OpenAI's method is the pass rule: a migration counts only when the outputs match exactly. You can build that check on your own code before you trust any model with it.

Collect real inputs, run the old build and the migrated program on each one, and compare the bytes:

import subprocess
import sys
from pathlib import Path

OLD = ["./legacy/payroll"]       # the existing COBOL build
NEW = ["python3", "payroll.py"]  # the migrated program

def run(cmd, data):
    result = subprocess.run(
        cmd, input=data, capture_output=True, timeout=60
    )
    return result.returncode, result.stdout

failed = 0
for case in sorted(Path("cases").glob("*.in")):
    data = case.read_bytes()
    if run(OLD, data) != run(NEW, data):
        failed += 1
        print(f"DIFF {case.name}")

print(f"{failed} failing cases")
sys.exit(1 if failed else 0)

It compares exit codes and stdout as raw bytes, so trailing spaces, line endings and padded numbers all count. If your batch job writes files instead of printing, compare those files the same way.

The cases that break migrations are the quiet ones. OpenAI's example is COBOL's fixed-point arithmetic, which truncates extra decimal digits unless rounding is requested (OpenAI). In Python, the decimal module lets you say which behaviour you mean:

from decimal import Decimal, ROUND_DOWN, ROUND_HALF_UP

amount = Decimal("2") / Decimal("3")
cents = Decimal("0.01")

truncated = amount.quantize(cents, rounding=ROUND_DOWN)
rounded = amount.quantize(cents, rounding=ROUND_HALF_UP)

Two thirds truncates to 0.66 and rounds to 0.67. A migrated program that reaches for float and round() takes the rounding path, and every test that only checks "close enough" will pass it. Seed your cases/ folder with values like these, plus empty records and maximum field widths. TokenPost makes the same point: changes to rounding or to how missing records are handled can alter a result even when the replacement appears to work.

Running the migration as a task

A COBOL port is long agent work: read the program, write the new one, run the parity script, fix the diffs, run it again. Give it a branch of its own so the legacy tree stays untouched while the agent iterates.

I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex and other command-line agents side by side, each in its own terminal and its own worktree (README). ⌘N starts a task on its own hermes/<name> branch, with the task as the agent's first prompt. That lets you hand the same migration ticket to two agents and run the parity script against both branches. I haven't run OpenAI's models on a COBOL migration myself.

Hermes IDE 2.3.4: Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch in the sidebar.

Hermes is free to download, source-available under BSL 1.1, and runs on macOS, Windows and Linux. It is not affiliated with OpenAI, Anthropic or any agent vendor.

OpenAI is not alone here: TokenPost points to IBM's COBOL-to-Java tool with comparison tests, and to Anthropic's published approach to migrating COBOL with Claude Code. Whichever agent you pick, the parity script decides whether the port ships.

What to do this week

If you own COBOL, Fortran or Visual Basic code, three steps cost little:

1. Build the parity harness and a cases/ folder from real production inputs, before any model touches the code.
2. Run MultiLangBench on the 272-test subset for the language you care about, if you want a number of your own rather than OpenAI's.
3. Try one small, well-tested program with GPT-6.1 Sol through the API, and count how many of your cases come back byte-identical.

Do you have legacy code you would let a model migrate today, if the parity harness came back clean? Tell me in the comments.

If this was useful

A migration like this is an agent loop: write, run the check, read the diff, try again, and stop when the harness is green. The AI Agents Pocket Guide covers the patterns behind that loop, from tool calling to guardrails that keep an agent from declaring victory early. If you want to run two agents on the same ticket, Hermes IDE is free and I build it.