Yes, your agency is almost certainly using AI to write part of your code, and that is fine. What is not fine is AI-drafted code that no human read carefully and no test exercises, merged on a Friday because the demo was due. The fix is not a tool ban. It is a short list of controls you put in the contract and the repository before the first sprint.
Why this matters more than it did two years ago
A developer with a coding assistant can produce a working-looking feature in an afternoon that used to take three days. That is real. The problem is that review, testing and understanding did not get faster at the same rate. If an agency keeps the same review habits it had before and triples its output, you get the same bug rate per line applied to three times as many lines.
Technical debt used to accumulate at human speed. Now it accumulates at machine speed, and it looks clean on the surface. The first version of our own outreach engine, described in section 4, was a five-step chain that read beautifully: every step well named, every function confidently commented, nothing a reviewer would stop on. It also had five places to fail silently, and the clean surface was exactly why nobody questioned the four extra steps. AI-written code is well formatted and well named, which is precisely what makes a weak reviewer skim it. We wrote about the general discipline in technical debt management. This post is about the specific version that shows up when a vendor is writing the code.
The five things to require
1. A test gate that blocks merges, not a testing promise
"We write tests" is a sentence. A gate is a configuration. Ask for both of these before work starts:
- Branch protection on main so nothing merges while the pipeline is red. GitHub documents this under protected branches, and every major host has an equivalent.
- A coverage threshold enforced in CI, not reported in a slide. On a TypeScript stack this is a few lines in a Vitest or Jest config. The number matters less than the fact that it fails the build when it drops.
Then ask one question that separates agencies: "Can you show me a pull request from last month that was blocked by a failing test?" If the answer is "our tests never fail", the tests are not testing anything.
2. A named human who reads every AI-drafted diff
Assistants produce code. They do not take responsibility for it. Your contract should name who does, and the repo should prove it: every pull request has a reviewer who is not the author, and the review has comments, not just a green checkmark.
Watch for two anti-patterns:
- Self-merged pull requests. One person prompts, one person approves, same person. That is zero review.
- Review by AI only. AI review bots catch style and obvious bugs. They do not know that your pricing rule has a VAT exception for Saudi customers. The human who knows the domain must read the diff.
3. A dependency and secrets policy
AI assistants love adding packages. They will reach for a library to pad a string. Each one is a supply-chain surface and a license question. Require:
- A lockfile committed and a rule that new dependencies are called out in the pull request description.
- No secrets, customer data or proprietary code pasted into tools that train on inputs. Most enterprise tiers of the major assistants offer a no-training setting; the contract should say the agency uses one.
- Clear assignment of all deliverables to you, however they were produced. Good agencies already do this. The AI clause just closes the "but a tool wrote it" loophole.
4. Fewer lines, not more
The cheapest code to test is the code that was never written. AI makes it painless to generate a six-step pipeline where one step would do, and every extra step is a place for a silent failure.
We hit this in our own outreach engine. It scrapes a prospect's site with a self-hosted crawler and uses a local model to draft one tailored email per company. The first version was a chain: extract facts, summarise, pick an angle, draft, polish. It was slower, pricier and, more importantly, it had five places to go wrong and five things to test. Collapsing it into a single call that extracts the facts and drafts in one pass beat the chain on both cost and output quality, and it halved the surface we had to cover with tests. We wrote the general case up in single call vs agent chains.
Ask your agency how they decide when generated code is too much code. If they do not have an answer, you will pay for the extra lines twice: once to write and once to maintain.
5. A codebase you could hand to someone else
The real test of AI-written code is whether a new engineer, with or without an assistant, can read it in a week. Require:
- Architecture decisions written down in the repo, in a short docs folder or ADR file, not in a chat history. Write down the why, not just the what. If your agency collapses a five-step chain into one call, as we did, the reason belongs next to the code, or the next engineer with an assistant will rebuild the chain, because a chain is what assistants reach for.
- Consistent patterns across features. AI assistants drift. Feature A uses one data access style, feature B another, because they were prompted on different days. Linting and a small set of templates fix most of this.
- Integration tests for the flows that make you money: signup, checkout, the webhook from your payment provider. Unit tests on helpers are cheap filler. The Saudi VAT exception from section 2 is the kind of rule that only an end-to-end test on a real checkout will catch, and covering the three paths like that with a tool like Playwright is what protects revenue.
What this does to the quote
Expect fair quotes to drop on boilerplate: CRUD screens, forms, admin tables, standard integrations. Expect them to stay roughly flat on the parts that were never typing-limited: the data model, the edge cases in your business rules, performance, and the review itself.
An agency whose quote dropped 60 percent across the board since adopting AI tools has either cut review or is about to. One whose quote did not move at all is probably not passing the savings through. The honest middle is a lower number on the predictable work and a line item for testing and review that did not shrink. If you are weighing engagement models, fixed price vs time and materials covers how to structure that.
Questions to ask on the first call
- Which AI coding tools do you use, and on what tier?
- Show me your branch protection and CI config on a recent project.
- Who reviews AI-drafted code, and can I see a pull request with real review comments?
- How do you decide when generated code is too much code?
- What happens to my code and data inside the tools you use?
Clear, specific answers mean the agency has thought about this. Vague reassurance means you are the first client to ask. Pair these with the broader software project red flags and you will screen out most of the risk in a single conversation.
The bottom line
AI coding tools are a speed multiplier. Multiply a disciplined team and you get a disciplined team that ships faster. Multiply a sloppy one and you get sloppy at scale, with a codebase that looks great until the first real incident. Your job as the buyer is not to pick the tool. It is to require the gate, the reviewer, the policy and the readable repo, and then check that they exist.
If you want a build where the tests are in the pipeline and the reviewer is named before the first commit, let's talk.
Originally published on the Pykero blog.
