The three-way comparison you actually need is not the one on the leaderboards
Pick any two of ChatGPT, Claude and Gemini and there is a benchmark where each one wins. That tells you almost nothing, because the benchmark isn't your codebase, your prompt, your latency budget, or your legal team's stance on data retention.
What does predict the outcome: how each product behaves when it hits the edge of what it knows, whether it can follow the seventh item in a ten-item instruction list, and whether a compliance review will approve the vendor at all. Those are things measurable in an afternoon with prompts already sitting in Jira.
This article is about running that afternoon.
Behavioural differences people consistently report
Treat everything in this section as a hypothesis to test, not a fact. Model behaviour shifts with every release, and the three vendors ship constantly. But these are the patterns developers describe over and over, and each one is checkable.
Long, multi-part instructions. Claude has a reputation for grinding through numbered constraints and often restating them before answering — helpful when you want to see what it thinks the job is, annoying when three lines of code were the goal. ChatGPT tends toward brevity and, under a long constraint list, is more often reported to quietly drop a later item. Gemini sits somewhere in between and sometimes compresses a multi-part request into a summary answer. Test: take a real ticket with eight acceptance criteria and count which criteria appear in the output.
Behaviour when it doesn't know. This is the single most expensive difference. ChatGPT is frequently described as producing a confident, plausible, wrong API call. Claude is more likely to hedge in prose — sometimes so much that the caveats have to be stripped. Gemini with search grounding enabled behaves differently from Gemini without it, which is worth knowing before any comparison. Test: ask all three to use a function that does not exist in your library and see who invents a signature for it.
Refusals and safety. The failure shapes differ operationally, not just tonally. ChatGPT typically returns a short refusal as normal content. Claude tends to explain and offer a partial answer. Gemini can block at the API layer, returning a response with no text and a finish reason indicating safety — which will throw an AttributeError in a parser that assumed response.text always exists. For pipelines processing user-generated content, that difference is a production incident waiting to happen.
Editing files versus writing them. Generating a new module from scratch is the easy case. The hard case is "change these three lines in this 600-line file and leave everything else alone." Some models return the whole file with silent unrelated edits. Some return a diff that doesn't apply. Tools like Aider expose different edit formats (whole-file, unified diff, search/replace) precisely because models differ here. Test with git diff --no-index on the before and after.
Agentic loops and tool use. Each vendor now ships a first-party coding agent: Claude Code, OpenAI's Codex, Gemini CLI. They differ in how many tool calls they'll chain before checking in, how they recover from a failed shell command, and how aggressively they read files before editing. The model and the harness are entangled — Claude inside Cursor is not Claude Code — so evaluate the combination that will actually ship.
Very long inputs. Every vendor advertises a large context window. Advertised capacity and usable capacity are not the same thing. Check the current documented limits directly, because they change, and then check the more important thing: paste an actual repo dump in and ask a question whose answer lives in the middle. That's where degradation shows up.
Characteristic failures. Reported patterns: ChatGPT invents plausible library APIs. Claude adds defensive code and explanatory comments nobody asked for. Gemini wraps JSON in markdown fences after being told not to. All three are fixable with prompting. Which one is cheapest to fix depends on the pipeline.
The harness
Install the three official SDKs:
pip install openai anthropic google-genai
export OPENAI_API_KEY=... ANTHROPIC_API_KEY=... GOOGLE_API_KEY=...
export OPENAI_MODEL=... ANTHROPIC_MODEL=... GEMINI_MODEL=...
Set the model env vars from each vendor's current model list — don't hardcode IDs into a script that'll outlive them.
bench.py:
import os, json, pathlib, random, concurrent.futures as cf
from openai import OpenAI
from anthropic import Anthropic
from google import genai
oai, ant, gem = OpenAI(), Anthropic(), genai.Client()
def chatgpt(system, user):
r = oai.chat.completions.create(
model=os.environ["OPENAI_MODEL"],
messages=[{"role": "system", "content": system},
{"role": "user", "content": user}],
)
return r.choices[0].message.content, r.usage.model_dump()
def claude(system, user):
r = ant.messages.create(
model=os.environ["ANTHROPIC_MODEL"],
max_tokens=8192,
system=system,
messages=[{"role": "user", "content": user}],
)
text = "".join(b.text for b in r.content if b.type == "text")
return text, r.usage.model_dump()
def gemini(system, user):
r = gem.models.generate_content(
model=os.environ["GEMINI_MODEL"],
contents=user,
config={"system_instruction": system},
)
# Gemini can return a blocked candidate with no text at all.
text = r.text if r.candidates and r.candidates[0].content else ""
return text, {"finish": str(r.candidates[0].finish_reason)}
RUNNERS = {"chatgpt": chatgpt, "claude": claude, "gemini": gemini}
def main():
cases = [json.loads(l) for l in open("cases.jsonl")]
out = pathlib.Path("runs"); out.mkdir(exist_ok=True)
manifest = {}
with cf.ThreadPoolExecutor(max_workers=9) as pool:
futs = {}
for c in cases:
for name, fn in RUNNERS.items():
futs[pool.submit(fn, c.get("system", ""), c["user"])] = (c["id"], name)
for f in cf.as_completed(futs):
cid, name = futs[f]
try:
text, usage = f.result()
except Exception as e:
text, usage = f"<<ERROR {type(e).__name__}: {e}>>", {}
d = out / cid; d.mkdir(exist_ok=True)
(d / f"{name}.md").write_text(text)
manifest.setdefault(cid, {})[name] = usage
# blind labels so you don't score the logo
for cid, vendors in manifest.items():
labels = ["A", "B", "C"]; random.shuffle(labels)
key = dict(zip(RUNNERS, labels))
for name, label in key.items():
(out / cid / f"{name}.md").rename(out / cid / f"{label}.md")
(out / cid / "key.json").write_text(json.dumps({"key": key, "usage": vendors}))
print("done ->", out)
if __name__ == "__main__":
main()
cases.jsonl comes from the backlog, not from a public eval set. Ten to twenty lines like:
{"id": "t-1041", "system": "You are a senior Go engineer.", "user": "Here is handlers/auth.go:\n<paste>\nAdd rate limiting per API key using golang.org/x/time/rate. Keep the existing error envelope. Do not change function signatures. Return only the changed functions."}
{"id": "t-1042", "system": "", "user": "Using our internal client, call billing.ReconcileInvoiceBatch(ctx, ids) and handle partial failure."}
That second case is deliberate — ReconcileInvoiceBatch doesn't exist. It measures hallucination, not correctness.
Score blind, one case at a time, five criteria, 0–2 each:
- Instruction adherence — every stated constraint honoured.
- Honesty — flags what it can't know instead of inventing it.
- Edit fidelity — no unrequested changes; patch applies.
- Format compliance — parseable on the first try, no stray fences.
- Signal-to-noise — shippable without deleting paragraphs.
Then compare outputs directly:
cd runs/t-1041 && git diff --no-index A.md B.md | head -60
Reveal the key file last. Brand priors are strong, including the scorer's.
For cost, don't guess: the usage dicts captured during the run give real token counts for real prompts. Multiply by each vendor's currently published rates. Do that in a spreadsheet that gets refreshed, because all three change pricing and tiering, and the cheap-tier models — Gemini Flash, the smaller ChatGPT and Claude tiers — often change the answer entirely.
The boring factors that actually decide it
Data retention and training. All three offer API terms that differ from their consumer chat terms, and the consumer ChatGPT, Claude and Gemini apps have their own opt-out settings that are not the same as the API defaults. Zero-retention arrangements exist but usually require asking. Read the current DPA for the specific product and tier being bought — not a blog post, not this one.
Region and deployment. Claude runs on Anthropic's API, AWS Bedrock and Google Vertex AI. ChatGPT models run on OpenAI's API and Azure OpenAI. Gemini runs on Google AI Studio's API and Vertex AI. Where EU-only processing is required, the answer is usually the cloud-hosted variant, and available regions per model are documented per platform. This constraint eliminates options faster than any capability test.
Rate limits. All three tier limits by account maturity and spend. A model that's fast in a notebook can throttle hard on launch day. Log the 429 responses and retry-after headers during the eval to see what's coming, and request increases before launch, not after.
Outages. All three have had them. Check status.openai.com, status.anthropic.com and the Google Cloud status dashboard. Build the fallback on day one: route through LiteLLM or a house adapter so switching vendor is a config change, and pick the fallback from a different vendor — a Gemini fallback for a Claude primary, not a smaller Claude for a bigger one.
Switching cost, honestly. The API call is the easy part. What binds you is everything around it: tool-call schema shapes, structured-output mechanisms, prompt-caching semantics (Anthropic's explicit cache_control blocks versus the other two's approaches), and — most of all — prompts tuned against one model's quirks. The eval suite is the real portability layer. If bench.py can be rerun and re-scored in an hour, switching is annoying. If it can't, switching is a quarter.
Where to start
If the bottleneck is multi-file refactoring in an existing repo, start with Claude via Claude Code, and test ChatGPT through Codex on the same three tickets — the edit-fidelity gap is real and it's measurable in one afternoon.
If the bottleneck is cost per call at volume on short, well-specified tasks, start with Gemini Flash, and test the smaller ChatGPT tier as the fallback — run both through the harness above with the actual token distribution before committing.
If the bottleneck is an existing Azure or GCP footprint with procurement as the long pole, start with whichever of Azure OpenAI or Vertex the org has already signed, and test Claude on Bedrock or Vertex as the second opinion — it can be added without a new vendor contract.
If the bottleneck is hallucinated APIs breaking the pipeline, don't start with a model. Start with the honesty case from cases.jsonl, run all three, and let the scores pick.
