Drawing the Line: What Deserves an LLM and What Doesn't

The three questions I use to decide what stays deterministic

Part 2 findings of an experiment: building an LLM-powered support agent with deterministic boundaries. The companion repo contains the full code.

Every AI feature reaches the same fork. This piece here — does it get a model, or does it get a method?

Go wrong one way and you've built a rules engine that can't read a sentence. Go wrong the other and a language model is deciding whether someone gets their money back.

"The AI interprets intent, software enforces policy" is easy to say — it's the principle this series started from. Applying it to a specific component on a specific Tuesday is the hard part.

This post is the ruler I use for that.

Three questions

For every component, in order:

1. Is the answer a fact or a judgment? Facts ("was the order paid?") have one correct answer. Judgments ("does this message sound angry?") tolerate several defensible ones.
2. What does a wrong answer cost? Nothing (a clumsy sentence), inconvenience (a wrong KB article), or money and trust (a refund to the wrong person).
3. Can a test pin the correct answer today? Not "eventually" — can you write a fixed-output assertion now?

flowchart LR
    A["New component"] --> B{"Is the answer a fact?"}
    B -- "no" --> AI["AI"]
    B -- "yes" --> C{"Costs money or trust?"}
    C -- "no" --> P["Either"]
    C -- "yes" --> D{"Can a test pin it today?"}
    D -- "yes" --> SW["Deterministic software"]
    D -- "no" --> HY["Software contract around AI"]
    classDef box fill:#eef2f6,stroke:#8fa3b8,color:#24313f
    classDef decision fill:#f7f4ec,stroke:#b3a988,color:#24313f
    classDef ai fill:#eef0f4,stroke:#8fa3b8,color:#24313f
    class A,AI,P,HY box
    class SW box
    class B,C,D decision

Applied to Post 1's components: intent interpretation goes to the AI. Refund eligibility to software. Retrieval targets end up hybrid (the interesting case, below). Refund execution is software plus a human gate. Policy exceptions are a job for a rule engine.

The interesting row is the hybrid

Retrieval deserves its own paragraph, because it's where most teams get tripped up. "Let the AI find the right knowledge base article" sounds like an AI decision. It isn't — at least not entirely.

The results are fuzzy: similarity search returns plausible articles, ranked, sometimes wrong. You cannot write assert(search("refund") == refundsArticle) and mean it.

But the call is rigid: which function runs, with what arguments, against which index. That part is plain software with a typed signature, and it's fully unit-testable:

// The results are probabilistic. The invocation isn't.
public interface KnowledgeBase {
    /** Returns up to k articles ranked by semantic relevance.
     *  Ranking quality is evaluated statistically, never asserted exactly. */
    List<KnowledgeArticle> search(Query query);
}

That split — fuzzy contents behind a hard contract — is what makes retrieval safe to hand to the model as a tool. The agent decides when to search; it never gets to redefine what searching means.

The meta-rule: even the line must be deterministic

Here's the part that took me longest to appreciate. Deciding who decides is itself a decision — so who makes that one?

If the answer is "the LLM classifies each action's risk tier at runtime," the whole architecture collapses: the boundary becomes another probabilistic output that can be talked into moving. Prompt injection doesn't need to break a rule if it can reclassify the action the rule applies to.

So in this system, the classification of every action is a static lookup — code, not judgment:

// dev/tonal/support/domain/RiskPolicy.java
public enum RiskTier { LOW, MEDIUM, HIGH, VERY_HIGH }

public final class RiskPolicy {

    private static final Map<ActionType, RiskTier> TIERS = Map.of(
            ActionType.SUMMARIZE_TICKET,    RiskTier.LOW,
            ActionType.DRAFT_RESPONSE,      RiskTier.LOW,
            ActionType.CLASSIFY_TICKET,     RiskTier.MEDIUM,
            ActionType.PROCESS_REFUND,      RiskTier.HIGH,
            ActionType.MODIFY_ORDER,        RiskTier.HIGH,
            ActionType.UPDATE_PERMISSIONS,  RiskTier.HIGH,
            ActionType.CANCEL_SUBSCRIPTION, RiskTier.VERY_HIGH,
            ActionType.DELETE_DATA,         RiskTier.VERY_HIGH);

    public static RiskTier tierFor(ActionType action) {
        return TIERS.get(action); // null = unclassified = fails closed
    }
}

Two deliberate choices in there:

• Map.of over clever logic. The tiers are policy, owned by humans, reviewed like any other business rule. A diff on this file should read like a changelog of governance decisions.
• Unclassified actions return null — and null fails closed. Adding a new action type without classifying it means the system refuses to act, not that it acts freely.

And the tests are about the map itself, not just lookups:

@Test
void everyActionMustHaveATier() {
    for (ActionType action : ActionType.values()) {
        assertThat(RiskPolicy.tierFor(action))
                .as("action %s must be classified", action)
                .isNotNull();
    }
}

@Test
void destructiveActionsAreNeverLowRisk() {
    assertThat(RiskPolicy.tierFor(ActionType.DELETE_DATA)).isEqualTo(RiskTier.VERY_HIGH);
    assertThat(RiskPolicy.tierFor(ActionType.CANCEL_SUBSCRIPTION)).isEqualTo(RiskTier.VERY_HIGH);
}

The first test is the important one: it pins exhaustiveness. Nobody can silently add an action type next sprint and forget to give it a risk tier — CI fails until they classify it. The boundary enforces itself.

Where the ruler bends (honestly)

• Facts can hide judgments. "Is the order within the return window?" is a fact — once you've decided what "within" means. Edge cases force policy writing, and policy writing is human work. Deterministic doesn't mean thought-free; it means the thinking happens once, in reviewable form.
• Costs shift with context. A misrouted ticket is cheap until it's routed to the legal team. Tiers are revisited as the system learns — the lookup table exists precisely so those revisions are one-line diffs.
• Judgments become facts with evidence. If eval suites eventually show some classification task measurably outperforming rules, moving it across the line becomes more of a defensible engineering. That's the measurable path we've mentioned last time.

Why this matters beyond support bots

The same three questions classify decisions in any domain where models meet consequences: clinical triage systems route judgment but never prescribe (fact, high cost); loan underwriting separates scoring models from disbursement logic; industrial safety controllers treat perception as input but interlocks as law. Wherever you look, the durable systems aren't the ones with the smartest model — they're the ones where nobody had to trust the model on a question that has a right answer.