Claude Code effort levels: what the A/B test really shows

Claude Code effort levels are supposed to be a dial you control, so it matters that people have started asking whether the level you pick is the level you actually get. A post by @argofowl on X, surfaced on Hacker News, claimed Anthropic was quietly reducing selected effort levels while showing users misleading numbers.

Anthropic replied. The reply is more interesting than the accusation.

🔍 What Anthropic actually said

Thariq, from the Claude Code team, posted the same response on X and in the HN thread:

"We sometimes test API serving configs in Claude Code before rolling them out, and one running now maps the numerical effort value differently. That's why Claude may tell some of you it's at '10' on high. The scale isn't 0-100, the number isn't meaningful on its own, and the effort you selected is the effort you're getting. We've run in-depth evals to confirm this doesn't affect model performance."

Read that carefully. It is not a denial of A/B testing. It is a confirmation of it, plus a claim that this particular test is behaviour-neutral. Both things can be true at once.

The original evidence was weak, and the thread said so. As commenter Wowfunhappy put it: the proof was that someone asked Claude what effort level it was set to, and "how would the model even know that?" Others argued effort is injected through the system prompt, so the model might genuinely see it. Nobody in the thread could settle it, which is exactly the problem.

📊 The real issue is unverifiable configuration

Strip out the outrage and what's left is a supply-chain question. You are paying for a product whose behaviour is set by three layers, and you can only see one of them.

The third row is where this story lives. A serving config can change on a Tuesday afternoon with no release note, no version bump, and no way for you to pin the old one. Commenter cube00 asked the sharpest question in the thread: "Why is it considered acceptable to test on paying customers without letting them know or giving them a way to opt out?"

Key takeaway: "Claude Opus 5, high effort" is not a specification. It's a label pointing at a moving target. Treat model behaviour as an external dependency with no lockfile, because that's what it is.

💰 Why over-thinking is a budget problem here

The complaint underneath the effort-level story is not that the model got dumber. It's that it got longer, and length costs money.

One commenter described asking for a config file update: on the older model it took under 2 minutes to read, parse and patch. On the newer one it ran 43 minutes, pulling containers, spinning up sandboxes, and writing test suites for the whole repo. Same single-file change at the end.

That's funny when your employer pays. It's not funny when you're a student or a two-person shop converting a USD subscription into rupees every month. A few things follow:

• Verbosity is the main cost driver, not model choice. One user (perching_aix) reported that comparing like-for-like, the majority of their cost overhead came from the bigger model simply being chattier, and that dropping to low reasoning beat falling back to a smaller model on quality-per-rupee.
• Higher effort is not free accuracy. Several developers in the thread independently landed on the same policy: medium by default, high only for genuinely hard architectural work, never the top tiers.
• Subscription tiers don't shield you. Limits are real. Burn your weekly quota on a task that needed two minutes of thinking and you've paid for it in lost days, not just tokens.

If you've never actually measured your prompt sizes, our AI token counter will show you what a given context costs against Claude, GPT and Gemini tokenizers before you send it. And if you're budgeting a USD subscription in rupees, the live LKR exchange rate is the number that actually decides whether the $200 tier makes sense for you.

🛠️ Run your own eval instead of arguing about vibes

Here's my honest position. I run agents against this codebase every day, so I have opinions about regressions, and I've noticed that almost all of them evaporate when I write the test down. Human memory of "it felt better last month" is worthless as evidence. So build a tiny personal benchmark. It takes an afternoon.

1. Pick 5 real tasks from your own repo that you already know the correct answer to. Not puzzles. Actual work: a bug fix, a refactor, a migration, a config change, a test.
2. Freeze the inputs. Same branch, same files, same prompt text, committed to git.
3. Log the outputs. Wall-clock time, tokens in and out, and a manual pass/fail you write yourself.
4. Re-run monthly, and any time you feel a regression.
5. Use the vendor's channel when you have data. Thariq explicitly asked users seeing a clear regression to file /feedback with the session ID, and offered credits. A session ID beats a complaint.

# crude but sufficient: one row per run, append-only
echo "$(date -Iseconds),opus5,high,$TASK,$SECONDS,$IN_TOK,$OUT_TOK,$VERDICT" \
  >> ~/agent-evals.csv

Five tasks, five minutes of logging each. That CSV is worth more than every "models are getting dumber" thread combined, because it's about your code.

💡 What this means for you

Anthropic's answer here was fast, named, and specific, which is better than the silence you'd get from most vendors. I don't think this particular test is evidence of deliberate degradation, and the accusation that triggered it rested on asking a model to introspect on its own settings, which is not a measurement.

But the structural point survives the debunking. The behaviour of a hosted coding agent is a dependency you cannot pin, cannot diff, and cannot roll back. That's a genuinely new category of risk for anyone shipping software, and it lands hardest on small teams without the budget to keep a second provider warm.

So: default to lower effort than you think you need, keep your prompts short because verbosity is where the money goes, hold a small eval suite you actually re-run, and keep one alternative agent configured so switching is a decision and not an emergency. None of that requires believing anyone is lying to you. It just requires treating the dial on your screen as a hint rather than a contract.