An Agent on a Leash, or why my AI agent doesn't make business decisions

This post kicks off the description of an experiment: building an LLM-powered support agent you can actually trust, one decision at a time. Everything described here ships in the companion repo reliable-ai-support (code as of this post: tag post-001).

The Support-Agent Problem

Picture this: It's 2 AM. A customer messages our support channel, frustrated because their recent order never arrived. They want a refund, or at minimum, an explanation. They've been emailing back and forth for three days with no resolution.

A support agent is assigned. But it's 2 AM. There's no one on call. So... does the AI just handle it?

That's the question I couldn't let go of — the one this whole series exists to answer.

The Great AI Illusion

The fantasy is seductive. Build an LLM-powered agent, give it access to the right tools, and let it handle issues end-to-end. It markets itself: "AI-powered 24/7 customer support!" Users love it. Everyone wins.

Except... what happens when the AI decides to process a refund to the wrong person? What happens when it accesses data it shouldn't? What happens when it "hallucinates" an order ID and tries to charge it?

The first thing I had to accept when designing this system: an LLM's confidence is not a reliability metric. It's a correlation metric. And correlation alone isn't enough.

The Boundary Problem

LLMs are great at understanding intent, surfacing information, writing responses. The problem is what happens when you hand them the keys to the kingdom.

So I built a table. A boundary that says "up to here, and no further."

The AI interprets intent. The software enforces policy. That's the split, and it's the thesis of everything that follows.

The Pipeline

If you remember one picture from this series, make it this one:

flowchart LR
    A[User input] --> B[Intent classification<br/>AI]
    B --> C{Deterministic validation<br/>software}
    C -->|low / medium risk| D[AI-assisted action]
    C -->|high risk| E[Human approval<br/>with audit trail]
    C -->|very high risk| F[Proposed only —<br/>human executes manually]
    D --> G[Execution or rollback]
    E --> G
    F --> H[Manual execution]

Every action passes through the deterministic gate first. What happens after the gate depends on the risk tier: low and medium risk actions proceed with AI assistance, high risk actions pause for human approval with a full audit trail, and very high risk actions are never executed by the system at all — the AI proposes, a human does the actual work. For now, remember just this: nothing executes without passing deterministic validation. No "let me just try it and see" energy allowed.

The gate either opens or closes based on hard-coded rules. The AI never holds the key.

The Autonomy Budget

I like to think of this as a budget. You have a certain amount of "AI agency" to spend, and once it's spent, the human takes over.

• Low risk: AI proposes, human disposes. Example: summarizing a support ticket, generating a response draft, highlighting relevant knowledge base articles. The AI does the heavy lifting, the human signs off.

• Medium risk: AI routes, human confirms. Example: directing a user to the right self-service option, suggesting the right KB article, classifying the ticket type.

• High risk: AI proposes action with audit trail, human approves. Example: processing a refund, updating a user's permissions, modifying an order. Money or critical data is involved.

• Very high risk: AI never executes. Proposes only. Human performs action manually. Example: canceling a subscription, changing a user's email address, deleting data. The AI can suggest what to do, but the actual action? That's on a human. Period. No exceptions.

The higher the risk, the more hands are in the cookie jar. And that's exactly how it should be.

The Honest Trade-offs

This design isn't free, and pretending otherwise would make this a sales pitch instead of engineering notes.

• Latency: high-risk actions now wait for a human. A refund that an agent could have executed in three seconds takes until someone reviews the queue. That's a real UX cost, and I pay it on purpose.
• Engineering effort: rule engines, approval queues, audit trails — all of this is code that "just let the AI decide" wouldn't need. The upfront cost is real.
• Rules can't capture everything: deterministic rules are only as good as the policy they encode. Edge cases ("the customer paid twice by mistake") don't fit neatly into yes/no checks yet.

What would change my mind? If eval suites get good enough that an AI's judgment on specific action classes is measurably more accurate than the rules — with error rates we can pin and monitor — then some of these gates could safely open. And building that evidence is exactly what I want this series to be about.

Back to 2 AM

So what actually happens to our insomniac customer now?

Their message hits the pipeline: intent classification understands "refund for undelivered order." The deterministic gate runs RefundEligibility.evaluate() — order not delivered, refund eligible. The system drafts a response, proposes the refund, files it in the approval queue with a full audit trail. At 8 AM, a human opens the queue, sees the proposal with every fact attached, and clicks approve. The customer gets their money back before their first coffee.

No one decided anything at 2 AM. The system prepared everything; a human decided something at 8. That gap between prepare and decide is the whole architecture.

Why This Matters to You

If you're building AI-powered systems, you've probably felt the tension between "move fast and break things" and "don't break customers' lives." This tension doesn't go away just because you're using fancier models.

The deterministic/non-deterministic split is where this series starts. As the system grows, other tensions will demand their own answers: what the agent's mistakes cost, how you test behavior that's different every run, which failures you design for versus prevent. Every trade-off gets documented — ADRs included — so you can watch the thinking evolve.

The autonomy budget translates directly beyond customer service bots: healthcare triage must route critical cases to physicians, financial chatbots can't move money without verification, autonomous vehicle perception must never execute without deterministic gatekeeping. The same pattern — AI interprets intent, software enforces policy — shows up wherever probabilistic intelligence meets real consequences.

The Boundary in Code: A First Look

The split between AI and software becomes concrete the moment you actually need to check whether a refund can happen. Here's the shape of the class that does it — a plain Java class living in the domain package, whose entire job is to answer one yes/no question: can this order be refunded under policy? Notice what's absent: no LLM call, no prompt, no model config. Just facts in, verdict out.

// dev/tonal/support/domain/RefundEligibility.java
public EvaluationResult evaluate(Order order, RefundRequest request) {
    // Deterministic business rules — no AI involved
    if (!order.delivered()) {
        return EvaluationResult.notEligible("Order must be delivered before refund");
    }
    if (!order.paid()) { /* ... */ }
    if (order.getAgeInDays() > RETURN_WINDOW_DAYS) { /* ... */ }

    // Eligible ≠ executed: actually issuing the refund goes through
    // a separate, risk-tiered authorization gate.
    return EvaluationResult.eligible(order.getId(), order.getCustomerId());
}

Two things worth noticing:

• eligible(...) is not an instruction to move money. It's a fact about policy. That comment is the whole thesis in two words: eligible ≠ executed.
• Same input, same verdict — always. The full class ships with unit tests that pin down this boundary (undelivered orders can never pass, eligible orders always pass), and it compiles and passes with no model configured. Try getting that guarantee from a prompt.

The complete implementation — records, all rules, and the test suite that proves the determinism claim — lives in the companion repo. For now, hold onto the shape: this is what "software enforces policy" looks like when it's not a slide.

And if you disagree with any of this — great. That means you're thinking critically about the problem. Leave a comment, let's hash it out.