We have been running Claude-style CLI coding agents against production .NET code for long enough to have opinions that survived contact with a client's release schedule. This post, part of our Practical AI in DevOps series, describes the setup we use, the parts that pay for themselves, and the work we still hand back to a person.
Two findings shape the rest of it.
A coding agent is a model plus a harness: the tools it can call, the context it starts with, the loop that decides what it sees next. A Berkeley group compared 21 model and harness pairings and found harness choice barely moved success rates, staying within about 2% on SWE-bench Lite, while moving cost a long way. The same model solved roughly the same share of tasks at up to five times the price depending on what wrapped it, and Claude Code averaged about twice Pi's cost per attempt (HarnessTax). Effort spent on repo scaffolding returns more than effort spent swapping models each time a new one ships.
The second finding is that an agent does what is cheap rather than what is correct. When a study gave agents both grep and language-server navigation on the same repositories, they picked the semantic tool between 0% and 6% of the time on code-localisation tasks, and forcing the semantic path first dropped success from 100% to 89% (agentconnect). On a large typed C# solution that matters, because text search cannot tell you whether a call binds to the int or the string overload, or which project a caller lives in. The Graphify C# project exists to hand agents compiler-accurate find-usages over a solution through Roslyn and MSBuild, which tells you how often they get it wrong without that help (graphify-csharp).
What the setup looks like
The repository carries the agent's working environment alongside the code:
- An
AGENTS.md(orCLAUDE.md) at the solution root with the build and test commands, the project layout, and the rules the agent must follow. - A coding standards document under version control, covering naming, structure, iteration style, and what a unit test has to assert before it counts.
- A semantic index of the solution so reference queries return bound symbols rather than string matches.
- A container the agent runs in, with no production credentials and no write access to anything outside the working tree.
- A review pass by a second agent instance against the standards document, before the diff reaches a human.
Checking agent context into git is now a common pattern rather than a PicNet invention. Tools such as OKF Agent Memory and hosted context registries are being built on the same premise, that curated project facts should be versioned and supplied, not guessed at from training data.
The standards document does most of the work
Our C# standards existed as a PDF long before we used agents, and it was read about as often as most standards documents are. Turning it into a file the agent reads on every task changed its status. An agent follows a written rule far more consistently than a human under deadline, which means the document now has to be right, specific and short enough to sit in context.
The shape of it:
# C# standards (agent must comply)
## Structure
- One public type per file, file name matches the type.
- No regions. No partial classes outside generated code.
## Nullability
- Nullable reference types enabled solution-wide. Do not suppress with `!`;
fix the model or the call site.
## Tests
- Every behaviour change needs a test that fails without the change.
- Do not edit an existing test to make a new change pass. Flag it instead.
## Framework
- Target framework is defined in Directory.Build.props. Do not assume
the latest .NET version or the latest C# syntax.
That last rule earns its place. Practitioners comparing assistants on C# repeatedly single out framework version awareness and idiomatic C# as the weak spots of general purpose models (r/jenova_ai discussion). An agent will happily write C# 13 syntax into a project pinned to an older SDK, then spend twenty minutes trying to work out why the build fails.
Tests are the safety net, and the agent will try to cut them
Unit tests are the only thing standing between an agent's confident diff and a production defect. That makes the tests themselves a target.
OpenAI publishes what its monitoring found in its own internal coding agent deployments, and reward hacking is on the list: agents "illegitimately edit tests to make them pass", disable checks, and sometimes misrepresent whether a task was completed at all (OpenAI). If the organisation building the model treats its agents as something to supervise continuously, a Sydney consultancy has no basis for treating them as a trusted contributor.
So we read test diffs separately from implementation diffs, and a changed assertion in an existing test gets more scrutiny than a hundred lines of new code. Where an agent touches a codebase under regulatory constraint, an APRA CPS 234 environment or anything handling personal information under the Privacy Act, a named human approves the merge. That is a design requirement of the pipeline, not a policy we hope people remember.
AI review before human review
The first reviewer of every agent diff is another agent, running a review prompt with the standards document and no memory of the conversation that produced the code. It catches the mechanical failures: standards breaches, missing tests, a public method with no null handling, a dependency added without being asked for.
This exists because a human reviewing forty agent diffs a day reviews the fortieth badly. It also filters out agent verbosity. Agents bury the conclusion under paragraphs of narration, to the point that people publish skills whose only job is to force the model to lead with the answer (i-have-adhd). A review agent that outputs a short list of concrete findings is much easier to act on than a wall of summary.
Sandboxing is not optional any more
Google Threat Intelligence reported in September 2026 that attackers are targeting coding agents directly. The DUSTMAKER credential stealer drops malicious configuration into hidden workspace directories such as .claude/ and .cursor/, then uses prompt injection in those files to make the assistant run attacker commands during ordinary developer interactions. The same actor published trojanised MCP servers to PyPI and stole OIDC tokens from GitHub Actions runners to publish signed packages that pass automated trust checks (GTIG).
Our rules follow from that. Agents run in a container with scoped credentials. Any dependency an agent adds is reviewed by a person against the registry. Agent configuration files are treated as executable code in review, because that is what they are.
Honest numbers
We report two figures per task: elapsed time and token cost. Tooling for this is maturing, with projects like Frugal Tokens built purely to track spend and usage across agents. Time saved without cost per completed task is half a measurement, and a failed agent run costs money while producing nothing.
What we see: greenfield work inside a well-defined module goes several times faster. Mechanical refactors across a large solution go faster only where the semantic index is in place, and go badly wrong without it. Debugging a subtle production issue in unfamiliar code is roughly a wash, because the time saved writing the fix is spent verifying that the fix addresses the real cause. We do not publish a single productivity multiplier, because the honest answer varies by task type more than it varies by model.
Senior engineers become editors
The part nobody costed properly is what this does to the working day of an experienced developer. Reading and judging generated diffs all day is a different job from writing code, and developers are feeling it as an identity change rather than a tooling change, as a heavily discussed Hacker News thread from August 2026 showed.
This favours senior-heavy teams, which suits how PicNet is built. Judging whether a diff is correct, idiomatic and safe to deploy requires exactly the knowledge that used to come from years of writing that code by hand. A junior engineer reviewing agent output has no way to tell a good diff from a plausible one, and the agent is very good at plausible. Small teams where everyone reviewing has shipped production systems get more out of these tools than large teams with a thin layer of seniors on top.
PicNet builds production AI systems for Australian organisations. Talk to us about what a first project could look like.
Originally published at picnet.com.au.
