Discuss+ The Hardest Thing an AI Agent Has to Learn Is When to Stop

Not how to do more. How to know when doing more is the wrong move.

A conversation about autonomous agents, task completion, unnecessary actions, and the surprisingly difficult question of what should happen after the work is done.

Approximately 22 min read

Part 1: For Everyone Who Has Ever Watched a Machine Keep Working on Something Nobody Asked It to Do

Imagine a washing machine that never knows when to stop

You put your clothes in the washing machine, choose a program, press Start, and go do something else.

At some point, the machine finishes. The clothes are clean. The water is drained. The program is complete.

What would you think if the machine continued washing the same clothes for another six hours?

You would not call it autonomous.

You would call a repair technician.

Now imagine a machine that does the opposite. It finishes the washing cycle, turns itself off, and never checks whether the water starts leaking ten minutes later. It does not notice that the next load has arrived. It never responds to a new problem because, according to its internal logic, the job is already done.

That machine is also not particularly impressive.

And somewhere between those two machines is a problem I have been thinking about while building an autonomous AI agent.

How does an agent know that its current task is finished, when to stop spending resources, and when it should start paying attention again?

Not just when to stop generating text.

Not just when to stop calling tools.

I mean when to stop working on a problem that no longer needs work.

And when to leave it alone without abandoning its responsibility to notice what happens next.

It sounds straightforward until you try to build a system that must make this decision repeatedly, without a human standing next to it.

Then the washing machine starts looking surprisingly intelligent.

We have spent a lot of time teaching AI to do things

Ask an AI agent to investigate a software bug. It reads files, searches for causes, edits code, runs tests, examines failures, and tries again.

Ask it to research a topic. It searches the web, compares sources, follows references, and assembles an answer.

Ask it to maintain a system. It checks the state, identifies problems, and perhaps takes corrective action.

These are useful capabilities. There is a lot of legitimate engineering involved in making them reliable.

But notice what these tasks have in common.

We tend to evaluate whether the agent can accomplish something.

Did it fix the bug?

Did the tests pass?

Did it find the answer?

Did it successfully perform the requested action?

Those are reasonable questions. However, they leave another question waiting in the corner:

Did the agent know when further action stopped being useful?

Imagine a coding agent fixes a bug, runs the tests, and gets a clean result.

What happens next?

A sensible system should verify the relevant acceptance criteria, report what it actually established, and finish the task.

An unsensible system might decide to refactor three nearby files because they look old, rewrite the test suite because it could be cleaner, or investigate a completely unrelated warning.

Technically, it is still doing work.

But the relationship between its work and the original goal has disappeared.

Congratulations. Your bug fix has become a small software migration, and your afternoon has left the building.

The problem is not necessarily that the agent is incapable of writing good code. The problem is that its ability to produce another action has outlived the reason for taking that action.

Being able to continue is not the same as having a reason to continue.

People stop working. Machines need a different mechanism.

Humans have several natural constraints. We get tired, hungry, distracted, and eventually need sleep. We move between activities. We receive new information. We have deadlines, responsibilities, and other things to do.

These mechanisms are far from perfect. People can spend four hours reorganizing their desktop instead of doing the important task. We can keep arguing after the discussion has become pointless. We can also stop too early because something becomes uncomfortable.

Human stopping behavior is not a solution we can simply copy.

An autonomous software agent has a different set of constraints.

It can keep running through the night. It does not have to go home, make dinner, or get some sleep before tomorrow's meeting. Depending on its architecture, it might perform checks for days without anyone opening a dashboard.

That sounds like a superpower.

It can also become an expensive way to achieve absolutely nothing.

A process can keep waking up, loading state, calling a model, rereading the same information, and reaching the same conclusion. It might keep attempting an operation that repeatedly fails. Worse, it might begin changing things that were already working.

The fact that a machine can operate continuously does not mean it should continuously perform new work.

A smoke detector stays responsible for detecting smoke even when there is no fire. It does not need to manufacture a fire to justify its existence.

That distinction is important.

A monitoring system may have a continuous responsibility without having a continuous need to intervene.

And this is where I think the discussion becomes more interesting than another debate about which model is smarter.

There are at least four different situations hiding behind the word "done"

Let's say an autonomous agent is responsible for maintaining a software service.

It detects a problem, investigates it, makes a permitted change, and runs the relevant checks.

What should happen when the tests pass?

There are several possibilities.

Situation one: The task is complete.

The required result has been achieved, and the available evidence satisfies the acceptance criteria. The agent should finish that task rather than keep modifying the result.

Situation two: The task cannot reasonably continue.

The agent lacks a required permission, a service is unavailable, two instructions conflict, or the available tools cannot establish the necessary fact.

It should stop the current attempt, preserve what it knows, and explain the blocker. It should not disguise an inability to proceed as success.

Situation three: Nothing needs to change right now.

The agent checks the relevant state and finds no condition requiring intervention.

This does not necessarily mean an unfinished task has been completed. It means there is no justified action in this cycle.

Situation four: Nothing needs to change, but something could change later.

The agent has finished its current check. It remains responsible for observing relevant events and responding if the situation changes.

These are not interchangeable outcomes.

A completed software audit and an idle monitoring cycle are different things. A blocked task and a healthy system with nothing to repair are also different things.

If an agent represents all of them with a single internal status called "DONE", we should probably ask some uncomfortable questions.

What exactly is done?

The task? The current execution? The monitoring cycle? Or the agent's entire responsibility?

Those are four very different promises.

The important distinction: finishing a task is not finishing your responsibility

This is the idea I keep coming back to.

An agent can be finished with a task without being finished with the responsibility that caused the task to exist.

Suppose an agent maintains a repository.

It checks the repository and finds no new issue. It does not need to modify anything. The current check is complete.

But that does not mean it should permanently forget about the repository.

A new commit might arrive. A dependency might change. A previously healthy process might fail. A security event might appear.

The correct response is not to keep repairing a system that has nothing wrong with it. Nor is it to declare the whole system permanently finished.

It is to stop unnecessary work while preserving the ability to react to relevant change.

That gives us a more useful way to think about autonomy:

A task has a result.
A system has an ongoing responsibility.
The current environment can change after a task is completed.
New evidence can justify a new task.

It sounds obvious when we describe it in ordinary language.

Implementing it reliably is another matter.

For example, how does the system know that a new event is relevant? Does every change justify another model call? What if the event is a duplicate? What if a previous conclusion depended on a file that has now changed? What if the system cannot establish whether the old conclusion is still valid?

We have moved from the simple question "Should the agent stop?" to something much richer:

What evidence is sufficient to finish a task, what conditions justify further action, and what must remain active after the task finishes?

That is a proper engineering problem.

Is anybody researching this, or are we inventing the washing machine all over again?

I went looking for research because I did not want to mistake an interesting idea for something nobody else had considered.

There is already substantial work on related problems, including some very recent studies. The important point is that the research does not give us a universal stopping rule for every autonomous agent.

It does give us useful pieces.

Research one: Knowing when to stop acting

In June 2026, Han Luo, Bingbing Wen, and Lucy Lu Wang published "Agentic Abstention: Do Agents Know When to Stop Instead of Act?" (https://arxiv.org/abs/2606.28733).

The authors studied tool-using agents across web shopping, terminal environments, and question answering. Their evaluation included 13 agent systems and two agent scaffolds across more than 28,000 tasks.

The crucial observation is that recognizing an impossible task and stopping promptly are separate challenges.

An agent might eventually discover that the requested item does not exist. But it may reach that conclusion only after several unnecessary searches and interactions.

In other words, it can get the final decision right and still take far too long to reach it.

The paper also reports that stronger models did not consistently perform better at timely abstention. More capability did not automatically translate into better restraint.

That is interesting because it challenges an assumption that often appears in discussions about AI agents: give the agent a more capable model, and it will naturally become better at knowing when to stop.

Apparently, not necessarily.

"Read the paper on arXiv" (https://arxiv.org/abs/2606.28733).

Research two: Knowing when not to act

A July 2026 paper, "AgentAbstain: Do LLM Agents Know When Not to Act?" (https://arxiv.org/abs/2607.10059), approaches the problem from another direction.

The authors evaluate 17 frontier models across four agent frameworks using 263 paired tasks in 42 sandbox environments.

Each pair contains two closely related situations. In one, the agent should act. In the other, a controlled change means it should abstain instead.

This detail matters.

An agent that always acts is not demonstrating good judgment. Neither is one that always refuses to do anything. The useful behavior is to distinguish the two situations.

The benchmark's paired-accuracy metric requires the agent to handle both sides of a pair correctly. The best model in the reported evaluation achieved 59.5% on that metric, while 13 of the 17 models scored below 50%.

Those figures belong to this specific benchmark. They are not a statement that agents are only 59.5% effective in all real-world work.

But the result illustrates something important: the ability to complete tasks and the ability to exercise restraint are different capabilities.

An agent that knows how to use a tool does not automatically know when using that tool is a bad idea.

"Read AgentAbstain" (https://arxiv.org/abs/2607.10059).

Research three: Proving that a task is complete

A newer August 2026 preprint, "When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs" (https://arxiv.org/abs/2608.23623), examines a narrower but closely related question.

What if an agent is not allowed to declare success merely because it says the job is done?

The proposed approach requires a completion certificate linking required claims to evidence in the execution trace. Derived values must also be reconstructable through a deterministic verification process.

In a reported controlled study, the method avoided premature unsupported completions that occurred in the comparison system.

That sounds promising, but the limits matter. The experiments use synthetic environments and narrowly defined task families. A certificate can establish that a result follows from recorded evidence under declared assumptions. It cannot magically establish that every external observation was true, that the original task specification was complete, or that the system is safe in every environment.

Still, the underlying idea is useful:

An agent's claim that it is finished should be something we can check, not merely something we can ask it to repeat more confidently.

"Read the preprint" (https://arxiv.org/abs/2608.23623).

So, have we solved it?

These papers show that the problem is being studied seriously. They also show that it breaks down into several smaller problems: recognizing infeasible tasks, avoiding unnecessary actions, stopping promptly, and supporting completion claims with evidence.

What they do not establish is a universal mechanism that can reliably determine when every long-running autonomous system should stop working, continue observing, or resume action.

That is a much broader goal.

And it is where I think there is room for a useful conversation between people researching agent behavior and people building systems that actually have to run.

Because the problem becomes especially interesting when the agent is not working on one finite assignment.

What if its job is to look after something continuously?

Part 2: For Developers, Programmers, and Anyone Who Has Ever Debugged a Loop That Refused to Die

Let's leave the washing machine alone for a moment and look at the architecture.

Imagine we have a long-running autonomous agent that periodically inspects a repository, evaluates possible changes, runs checks, and decides whether intervention is justified.

We are not building a one-shot chatbot response. We are building a system that maintains state across cycles.

At this point, a naive implementation might be something like:

Inspect the environment.
Ask the model what to do.
Execute the action.
Check the result.
Repeat until the model reports completion.

For a prototype, this is a reasonable starting point.

For an unattended system, it leaves a dangerous amount of responsibility with a single decision: whether the model thinks it is done.

What if the model is uncertain? What if it repeats itself? What if it finds no useful action but interprets that as an invitation to explore unrelated work? What if it reports success after a tool failed? What if the environment changes immediately after the result is verified?

We need to separate the parts of the control problem.

Do not confuse task state with runtime state

One design mistake would be to put every possible condition into a single enumeration and hope that the names explain the behavior.

For example:

"RUNNING", "COMPLETE", "BLOCKED", "NO_ACTION", "OBSERVE", "REOPEN".

These labels are useful, but there is a hidden problem. They do not necessarily describe the same dimension.

"COMPLETE" describes the outcome of a task.

"BLOCKED" describes why execution cannot currently proceed.

"NO_ACTION" describes a decision not to perform a change in the current cycle.

"OBSERVE" describes a runtime posture.

"REOPEN" describes a transition triggered by new information.

Trying to represent all of these as equivalent states can make the implementation confusing.

I would start by separating at least two dimensions.

Dimension A: What happened to the task?

State| Meaning
"COMPLETE"| Required conditions have been met and verified.
"INCOMPLETE"| Work remains, but it is still reasonable to continue.
"BLOCKED"| A necessary input, permission, capability, or external condition is unavailable.
"FAILED"| The attempt ended unsuccessfully under a defined failure condition.
"NEEDS_REVIEW"| The system cannot safely resolve a decision without escalation.

These states should mean something operational. They should not be labels that a model can select without consequences or evidence.

Dimension B: What should the long-running service do now?

Mode| Meaning
"ACTIVE"| Execute an approved task or continue useful work.
"OBSERVE"| Monitor for relevant change without making interventions.
"WAIT"| Wait for a known event, scheduled check, or external dependency.
"STOPPED"| The service itself has been shut down or disabled.

A task can be "COMPLETE" while the service remains in "OBSERVE".

A task can also be "BLOCKED" while the service continues monitoring for the missing dependency to become available.

Those combinations are not edge cases. They are normal situations in a long-running system.

A third mechanism can handle reopening or scheduling new work when an event invalidates an earlier conclusion.

This is a conceptual architecture, not a claim that these exact states have already been implemented or validated in Sentinel.

The important design principle is that task completion should not automatically determine the lifecycle of the entire service.

A function returning "true" should not secretly mean that the whole autonomous system can forget its responsibilities.

Define completion before asking the agent to achieve it

Let's take a coding task:

"Fix the authentication bug."

That is not a sufficiently precise completion condition for an unattended agent.

The code might compile while the vulnerability remains. A regression test might pass while a different authentication route stays broken. The agent might change a test to make it pass without fixing the underlying issue.

We need an explicit definition of success.

For example:

The specified vulnerability no longer reproduces.
The required regression tests pass.
Existing relevant tests do not regress.
The change remains within the permitted scope.
The final state can be inspected and verified.

The exact criteria depend on the task. A monitoring cycle may require no code change at all. A deployment task may need a verified health check. A data migration may need consistency checks against both the old and new state.

The key is that the criterion is designed before the system reaches its conclusion.

Otherwise, the agent can start with a vague goal, work until it feels satisfied, and invent the explanation afterwards.

That is a recipe for a very confident progress report.

Anthropic's "Demystifying evals for AI agents" (https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) discusses the combination of deterministic checks, model-based evaluation, and human judgment. One important lesson is that different parts of an agent's performance require different kinds of verification. Code-based graders can check specific outcomes reproducibly, while model-based graders can handle more nuanced criteria but introduce their own uncertainty.

We should apply that principle to termination.

If a required condition can be checked deterministically, do not replace the check with a model's opinion about whether the condition probably holds.

If a requirement is subjective or inherently uncertain, state that limitation and define how the result will be reviewed.

A task should not become "COMPLETE" simply because the next action seems boring.

A successful execution trace is not the same as a successful task

Consider this sequence:

The agent calls a deployment tool.

The tool returns an acknowledgement.

The agent reports, "Deployment successful."

But did the deployment actually complete? Is the new version healthy? Did the request reach the intended environment? Was the acknowledgement merely a receipt that the request had been queued?

The difference between an action being accepted and an intended outcome being achieved is fundamental.

This matters even more when the agent performs several steps. A successful final command does not prove that every earlier assumption was correct.

A useful completion record might capture:

The task identifier and version of its acceptance criteria.
The required conditions and their verification results.
The relevant evidence for each condition.
The state of important external side effects.
Any remaining uncertainty or unmet requirements.
The reason the task was permitted to terminate.

Not every task needs an elaborate certificate. A trivial operation might only need one deterministic check. A consequential task may need independent verification and a more structured record.

The point is not to create paperwork for its own sake.

The point is to distinguish a real, verified outcome from a plausible narrative about an outcome.

The August 2026 preprint "When May an Agent Stop?" (https://arxiv.org/abs/2608.23623) explores one version of this approach through evidence-carrying termination. Its experiments are narrow and synthetic, so I would not treat the reported results as proof of general reliability. But the architectural question is worth taking seriously: can the runtime reject a completion claim when required evidence is missing or inconsistent?

An important limitation remains.

A verifier can establish that a claim follows from the evidence it receives. It cannot guarantee that the evidence accurately represents reality if the underlying tool, sensor, or source is unreliable.

Good termination therefore needs both a valid task specification and trustworthy-enough verification.

No action is a legitimate result, not a failure to produce output

Let's say the agent runs a security inspection.

It finds no new relevant change, no rule violation, and no condition requiring intervention.

The expected outcome is not necessarily another action.

It may be a verified observation that there is currently no reason to act.

That sounds trivial, but consider what happens if the architecture treats every cycle as an instruction to make progress.

Progress toward what?

If the system has no remaining task, it may start finding work that was never requested. In a codebase, this could mean gratuitous refactoring. In a cloud environment, it could mean changes to settings that do not need changing. In a security tool, it could mean reacting to harmless events.

The mechanism that makes an agent useful, its ability to choose actions, can become a source of unnecessary side effects.

One practical solution is to allow a cycle to finish with an explicit reason for no action.

For example:

"NO_ACTION: no relevant state change detected"

Or:

"NO_ACTION: candidate did not meet the intervention threshold"

Or:

"NO_ACTION: evidence insufficient to justify a mutation"

These examples describe different situations. The details matter because a generic "NO_ACTION" can otherwise hide very different causes.

One means the system checked and found nothing requiring a change.

Another means it found a candidate but rejected it.

The third means it could not establish that an intervention was justified.

The last case may require additional observation or human review, depending on the risk. It should not automatically be counted as a successful check.

We need to be precise about what "nothing happened" actually means.

Did we inspect the relevant state and establish that intervention was unnecessary? Did we fail to obtain the necessary information? Or did we simply decide not to run a check?

Those are not equivalent outcomes.

A quiet log is not proof of a healthy system.

Observation needs its own explicit semantics

This is where the discussion becomes particularly relevant to agents that run continuously.

Imagine a monitoring agent that checks a service every few minutes.

At 10:00, it verifies that everything is healthy.

At 10:05, nothing relevant has changed.

Should it rerun its entire analysis, or can it reuse the previous conclusion?

That depends on what the previous conclusion actually covered and what has happened since then.

If the state relevant to the conclusion remains unchanged, repeating the same expensive investigation may provide little value.

If an input changes, the previous result may no longer apply.

A sensible architecture should distinguish:

Verified current state: a conclusion supported by the latest relevant evidence.
Unchanged state: no relevant event has invalidated that conclusion.
Uncertain state: the system cannot establish whether the old conclusion remains valid.
Changed state: new evidence requires an evaluation or a new task.

An observation cycle should not silently transform uncertainty into confirmation.

Suppose a scanner cannot read one of its input files. It should not reuse an old "no issue found" decision as though the file had been inspected successfully.

Likewise, if a local dependency has changed, a previous decision about the importing module may no longer be valid, even if that module's own source has not changed.

This is a general cache-invalidation problem as much as an agent problem.

The system must know what its conclusion depends on and what changes can invalidate it.

That does not require the model to reconsider the entire environment every time. It requires the runtime to track the conditions under which previous conclusions remain valid.

The subtle difference between sleeping and observing

In ordinary software, an idle process might wait for an event rather than repeatedly poll an expensive dependency.

An autonomous agent can use the same idea.

When there is no justified work, it can:

Wait for a known event.
Schedule a bounded future check.
Monitor a narrow set of signals.
Reevaluate only the parts of the state that have changed.

This can reduce unnecessary computation without requiring the service to abandon its responsibility.

However, an observation mechanism must define its blind spots. If the event source is incomplete, events can be lost. If the polling interval is too long, a change may go unnoticed. If the system watches only a narrow set of signals, it may miss a relevant event outside that scope.

Observation is not magic. It is an explicit trade-off between cost, responsiveness, and coverage.

For a security agent, that trade-off should be documented and measured rather than left to intuition.

Reopening a completed task requires a reason

Suppose the agent has verified that a module is healthy.

Ten minutes later, its dependency changes.

Should it reopen the task?

Potentially, yes.

But what exactly changed? Does the dependency affect the module? Does the old verification cover the new version? Is the change relevant to the original risk?

Without a dependency between evidence and conclusions, the system has two bad options.

It can repeat everything on every cycle, wasting resources.

Or it can reuse stale conclusions after the assumptions that supported them have changed.

A better approach is to define what invalidates a result.

A conclusion may depend on file contents, dependency versions, configuration, permissions, external service state, or the version of the rules used to evaluate the task.

If those dependencies change, the system can decide whether to invalidate the entire result or only the affected portion.

This introduces an important distinction between a new event and a new reason to act.

A file modification is an event.

A file modification that invalidates a security assumption is a reason to reevaluate.

A reevaluation is not automatically a reason to mutate the system.

The agent still needs to inspect the new state, decide what the change means, and justify any intervention.

That gives us a chain of decisions rather than a single "wake up and do something" trigger.

It also gives us a more useful way to handle recurring tasks:

Reopen the smallest piece of work that the new evidence actually invalidates.

The exact scope depends on the system's dependency model, and there will always be cases where the impact cannot be determined reliably. In those cases, the runtime should expose the uncertainty instead of silently treating the old conclusion as current.

Hard limits are necessary, but they do not define success

There is a practical reason many agent frameworks provide maximum iterations, timeouts, and tool-call budgets.

Without a hard limit, an agent loop can continue indefinitely.

Microsoft's "Agent Looping documentation" (https://learn.microsoft.com/en-us/agent-framework/agents/looping) explicitly warns developers to bound autonomous loops because completion conditions can fail, models can stall, and evaluators may be probabilistic. The feature is also described as experimental.

That is sensible advice.

A runtime can enforce limits on:

Maximum steps.
Maximum elapsed time.
Maximum tool calls.
Maximum model or token spend.
Repeated identical actions.
Consecutive cycles without meaningful progress.

But these are safety rails, not evidence of completion.

If an agent hits its maximum number of attempts, that does not mean the task is complete. It means the execution budget has been exhausted.

If a tool times out, that does not mean the desired operation failed in the external world. Its outcome might be uncertain.

If the agent repeats an action three times, the correct response depends on whether the attempts were meaningfully different and whether new information justified them.

A bounded runtime should return an honest terminal reason.

For example:

"BLOCKED: dependency unavailable after the retry budget"

Or:

"NEEDS_REVIEW: external outcome could not be verified"

Or:

"FAILED: no progress after the allowed number of attempts"

The precise names are less important than the distinction.

A hard budget prevents runaway execution. It does not substitute for a task-specific completion criterion.

We should not confuse "the process stopped" with "the task succeeded."

The hardest part might be defining progress

An agent can be active without making progress.

It can read the same file repeatedly, issue slightly different search queries, retry an unchanged request, or rewrite a plan without changing the state of the world.

The logs can look busy. The token counter can rise. The tool-call count can increase.

None of those metrics, by themselves, establishes that the task is getting closer to completion.

For iterative work, we need a definition of meaningful progress.

Depending on the task, that might mean:

A previously unmet acceptance criterion is now satisfied.
New evidence has resolved an uncertainty.
A failing test has moved to a passing state.
A verified artifact has been created.
An obstacle has been removed.
The set of viable next actions has changed based on new information.

Some tasks are difficult precisely because progress is not linear. A search may return nothing useful, yet establish that a particular route is exhausted. A failed test may identify a previously unknown constraint.

Therefore, a simple rule such as "stop after three steps without an artifact" can stop legitimate work too early.

A better design uses multiple signals and evaluates them in the context of the task.

For example, repeated identical tool calls are strong evidence of an unproductive loop unless something relevant has changed. A sequence of different experiments that systematically eliminates hypotheses may still be productive even if no final artifact exists yet.

This is why progress detection should not be reduced to one universal counter.

It also needs an exit condition for the case where the agent has exhausted the available approaches without reaching the goal.

Failure to make progress should trigger a decision, not automatically trigger another attempt.

How do we test a decision to stop?

This is where developers can make the problem measurable.

We can build tests that ask whether the system acts when it should and refrains when it should not.

A one-sided benchmark is dangerous.

If we only test situations where an agent should act, we can build a system that intervenes aggressively on every possible signal.

If we only test situations where it should remain idle, we can build an agent that never accomplishes anything.

The benchmark needs both kinds of examples.

Anthropic's "agent evaluation guidance" (https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) discusses the same general issue: evaluations must cover both desired behavior and cases where that behavior should not occur. Otherwise, optimization can create a system that overuses a capability because the evaluation never penalizes unnecessary use.

For termination, I would begin with a test matrix like this:

Scenario| Expected behavior
All required conditions are verified| Complete the task.
The result looks plausible but a required check is missing| Do not claim completion.
A required dependency is unavailable| Stop or escalate with an explicit reason.
An action has already failed and nothing relevant has changed| Avoid an unjustified identical retry.
New relevant evidence invalidates the previous conclusion| Reevaluate the affected task.
An unrelated file changes| Do not automatically reopen every task.
A monitoring cycle finds no reason to intervene| Finish the cycle without a mutation.
The observation source fails| Record uncertainty rather than claiming the system is healthy.
The action budget is exhausted| Stop the execution, but do not label the task successful.
The environment is healthy now but may change later| Finish the task while preserving the required monitoring behavior.

The point is not to create ten hand-picked tests and declare victory.

We need a benchmark that exercises the boundaries between the states, including cases that differ by only one relevant condition.

For instance, take two nearly identical scenarios. In the first, a dependency has not changed. In the second, it has. The correct response may be to reuse the old decision in the first case and invalidate it in the second.

That test examines something much more meaningful than whether an agent can produce a plausible explanation for its choice.

It checks whether the runtime distinguishes the two situations.

The July 2026 "AgentAbstain benchmark" (https://arxiv.org/abs/2607.10059) uses paired tasks to test whether agents can both act when appropriate and refrain when conditions call for abstention. That is a useful pattern for evaluating autonomous decision-making beyond task success alone.

The important caveat is that a benchmark remains a model of the world. Results from synthetic or sandbox tasks do not automatically predict performance under every production condition.

We need controlled tests, realistic scenarios, and eventually evidence from real usage.

Which metrics would tell us whether the design actually works?

I would avoid relying on a single number called "autonomy."

It is too vague to tell us whether the system is completing tasks, wasting resources, stopping too early, or neglecting monitoring.

Instead, we can measure distinct failure modes.

Premature completion rate

How often does the agent declare a task complete while one or more required conditions remain unmet?

Unnecessary action rate

How often does it execute a mutation or tool call when the evidence supports no intervention?

Not every extra read is harmful, so this metric should distinguish harmless inspection from risky or costly action.

Delayed stopping

When sufficient information becomes available to stop, how many unnecessary interactions occur before the agent reaches that decision?

This is close to the problem examined in the June 2026 Agentic Abstention paper.

False reopening rate

How often does an irrelevant event cause a completed task to be reopened?

Missed reopening rate

How often does a relevant change fail to invalidate an old conclusion?

This is particularly important for an agent that uses stored decisions or cached results.

Unverified completion rate

How often does the system report success without sufficient evidence to support its completion claim?

Cost per verified outcome

How much computation, time, and external tool usage are required to reach an outcome that actually satisfies the task's conditions?

Observation coverage

How reliably does the system detect relevant changes while it is not actively intervening?

This needs careful interpretation. A low action rate is not evidence of good observation, and a high action rate is not evidence of comprehensive monitoring.

These metrics can conflict.

A more conservative agent might reduce unnecessary actions but increase the number of tasks it sends for human review. A more aggressive agent might complete more tasks but produce more unwanted changes.

The goal is not to minimize every number independently.

The goal is to establish acceptable trade-offs for a defined task and risk level.

For a low-risk information-gathering task, an extra search may be harmless.

For a production security agent, an unnecessary configuration change can be far more expensive than a delayed intervention.

The evaluation should reflect that difference.

Completion evidence has a boundary too

There is one more problem.

Even a carefully designed verifier cannot prove more than its assumptions allow.

Suppose an agent runs a suite of tests and all the tests pass.

That establishes something useful: the implementation passed those tests under the conditions in which they ran.

It does not prove the absence of all defects.

Suppose an external monitoring service reports that an endpoint is healthy.

That provides evidence about the endpoint according to that monitoring service's checks. It does not prove that every component behind the endpoint is correct.

Suppose an agent generates a completion certificate. If the original acceptance criteria omitted an important requirement, the certificate can be internally valid while the overall task remains inadequate.

This is not a reason to abandon verification.

It is a reason to state what has actually been verified.

The August 2026 preprint on "Evidence-Carrying Termination" (https://arxiv.org/abs/2608.23623) is especially relevant here. Its authors explicitly limit their claims: their approach establishes trace support under declared assumptions, not universal task success, external truth, safety, or alignment. The work uses synthetic task environments, so further validation is needed before generalizing its results.

That qualification is important.

A reliable system should not turn a limited check into an unlimited claim.

For example, "All required tests passed" is more defensible than "The software is correct."

Likewise, "No actionable event was detected by the configured monitors during this cycle" is more informative than "Nothing happened."

The second statement in each pair hides the boundary of the evidence.

The first makes that boundary visible.

The runtime and the model should not have identical responsibilities

Here is a design principle I would like to see more often in autonomous systems.

The model can propose what to do next, explain its reasoning, and identify evidence relevant to a task.

The runtime should enforce the limits and conditions that determine whether the proposed action is allowed.

That separation matters because the model is part of the system being controlled. Asking the model to decide whether its own behavior satisfies every rule can leave the enforcement mechanism dependent on the same uncertain judgment it is supposed to evaluate.

For example:

The model proposes that a task is complete.
The runtime checks the mandatory acceptance conditions.
The runtime verifies that the evidence refers to the correct task and current state.
The runtime checks whether a relevant event invalidated the result.
The runtime accepts completion, requests further work, or stops with an explicit unresolved condition.

The model remains useful, especially when interpretation is required. But it does not get to redefine the conditions of success simply because it has reached a plausible conclusion.

The same principle applies to budgets.

The model can propose another attempt. The runtime decides whether that attempt is permitted under the task's remaining budget, retry policy, and safety rules.

The model can identify a possible opportunity for improvement. The runtime still needs a reason to authorize the intervention.

This is not a claim that all decisions can be made deterministically. Some tasks require judgment. But the system should make the boundary between judgment and enforceable rules explicit.

Anthropic's "Building Effective AI Agents" (https://www.anthropic.com/engineering/building-effective-agents) recommends starting with the simplest architecture that meets the need and adding complexity when the evidence justifies it. That is a sensible principle here too. A stopping mechanism should not become an elaborate multi-agent committee if a deterministic check and a clear runtime policy are sufficient.

We do not need more moving parts merely to announce that the machine has stopped.

We need the smallest design that can distinguish the states that matter.

What I would try in a real autonomous system

If I were designing this from scratch, I would resist the temptation to begin with a large collection of new status fields.

I would first define a small set of observable behaviors.

Can the system distinguish a verified result from an unsupported completion claim?

Can it finish a cycle without modifying anything?

Can it explain why it did not act, without pretending that every no-action decision represents success?

Can it preserve a monitoring responsibility after completing the current task?

Can it reopen only the work invalidated by a relevant change?

Can it stop when further attempts have no reasonable prospect of helping, while preserving enough information for a human or a later run to continue?

And can we test each of these behaviors independently?

I would then build a small set of replayable scenarios with controlled changes.

For example, the same task could run against two otherwise identical repository snapshots. In one snapshot, the relevant dependency remains unchanged. In the other, it changes. We would expect different invalidation behavior, while ensuring that unrelated changes do not trigger unnecessary work.

Another scenario could give the agent a successful-looking result with a missing verification step. We would test that it refuses to report verified completion.

A third could present an environment where no intervention is necessary. We would test both that it avoids unnecessary mutations and that the monitoring responsibility remains active.

Finally, I would introduce uncertainty deliberately. Remove a required input, make a tool unavailable, or provide conflicting evidence. We should see a clear blocked or review-needed outcome, not a fake success and not an endless retry cycle.

The results should be measured separately. A system can be excellent at avoiding unnecessary actions and terrible at detecting relevant changes. We should not allow success in one category to hide failure in another.

I would also treat simulated tests and production observations as different kinds of evidence. Passing a controlled test demonstrates that the behavior works in that test environment. Production evidence shows how the complete system behaves amid real timing, changing dependencies, service failures, and unexpected input.

Neither replaces the other.

And I would keep the implementation small until the results demonstrate that additional complexity is necessary.

No new memory field should exist merely because we can think of a plausible future use for it. No extra agent should be introduced merely because the architecture diagram looks more impressive with another box.

Every mechanism should have a defined purpose, an observable effect, and a way to test whether it helps.

That includes the mechanism responsible for knowing when to stop.

The Question I Keep Coming Back To

We often describe autonomy as the ability to take actions without waiting for a human.

But that definition is incomplete.

A system that acts without a human while lacking reliable boundaries is not necessarily meaningfully autonomous. It may simply be capable of producing side effects without immediate supervision.

A more useful question is whether it can operate within a defined responsibility, recognize when that responsibility has been satisfied for the current task, and respond appropriately when the world changes.

That involves several different capabilities:

Knowing what outcome was requested.
Establishing what evidence is sufficient to verify that outcome.
Recognizing when further action is unlikely to help.
Avoiding unnecessary intervention.
Preserving the ability to observe relevant changes.
Resuming work when those changes justify it.
Stopping safely when uncertainty, limits, or risk prevent further progress.

I do not think a single prompt or a single model upgrade can guarantee all of this.

The research discussed above points toward different parts of the problem, from timely abstention to evidence-supported completion. Those are valuable advances. But each addresses particular settings, and the results need to be interpreted within their experimental limits.

For builders, that means defining acceptance criteria, separating task outcomes from runtime behavior, creating balanced evaluations, and measuring both unwanted action and unwanted inaction.

For everyone else, the intuition is simpler.

A good washing machine finishes the cycle.

A good smoke detector does not start a fire to justify its existence.

And a good autonomous agent should not manufacture work merely to prove that it is still autonomous.

Sometimes the correct next action is no action. Sometimes the correct next action is to keep observing. And sometimes the most important thing an agent can do is recognize the difference.

The difficulty is not writing those three sentences.

The difficulty is building a system that can distinguish those situations reliably, demonstrate why it made that choice, and do so repeatedly without turning every quiet moment into another expensive investigation.

What do you think?

I'd genuinely like to hear from people who build agents, evaluate models, work on control systems, or simply spend their days trying to make software behave predictably.

What should count as sufficient evidence for an autonomous agent to declare a task complete?

Would you use an external verifier, a collection of deterministic conditions, a model-based judge, or some combination of all three?

And for a long-running agent, where would you draw the line between "the task is finished" and "the system should remain responsible for watching what happens next"?

I am especially interested in examples where an agent either kept working after it should have stopped or stopped when it should have continued observing.

I suspect the most useful answers will come from the failures we can reproduce and measure, not from another promise that the next model will simply know better.

Because the goal is not to build an agent that never stops.

It is to build one that knows why it is continuing, why it is stopping, and what would justify changing that decision.

And if we manage that, perhaps we can finally leave the washing machine alone.