For over a decade, we religiously used the well architected framework (WAF) in design reviews. Objective assessment with clear guidance from WAF made our designs risk free (or risk managed) with ambiguities and gaps called out. With agentic AI, there is always a little extra ambiguity. The rhythm of WAF does not strictly match one particular assumption underneath agentic AI: that we can diagram the execution path before the request arrives. All new ambiguities generally stem from this one root.
I’ve run design reviews for more than a decade now. Amazon retail first, then AWS, then my own startup, and now healthcare. The rhythm never varied much. Scrutinize the design against the WAF pillars, weigh it against the alternatives, name the gaps and the risks and the open questions, turn the trade-offs into decisions, then move and manage what’s left in the risks. Whether we follow AWS’s six pillars, Google’s, or Microsoft’s four, the content is close enough that the muscle memory transfers. We are answering one question in six different registers: is this system built well enough to trust?
And every design I reviewed in those years shared a property so basic that stating it sounds silly. We could enforce the execution path in advance. A single user request might fan out across dozens of services (or a few hundred in retail), queues, and databases, but an engineer could still draw the expected sequence diagram, project TPS for every service, point out the failure modes (and single points of failure), and estimate how much system stress one request would generate.
Our early ML workloads (or traditional ML) fit that mold too. Request comes in, features go into a model, inference comes out, and the surrounding application decides what happens next. The model was a component with a latency budget, not a decision-maker.
What actually broke, or started to smell
Let me be precise about what did not break, because this gets muddled and debated constantly.
The wire protocols are fine. MCP and A2A are still request-response, and we can secure, rate-limit, and observe them the way we always have. The individual components are fine too. A model call, a tool invocation, an API request: all of these harden under the existing framework without much reinterpretation.
The break happens one layer up.
By design, one request to an agentic service becomes a variable number of model calls, memory retrievals, tool invocations, and inter-agent messages. In some designs that number is unbounded. At each iteration the model decides what to do next, which tool to reach for, whether the result is good enough, and whether another reasoning cycle is warranted. The same input can take a different path tomorrow. Add persistent memory and a request that looks isolated also depends on everything the agent retained from sessions we weren’t watching.
So the call graph is no longer fully configured before deployment. Part of it gets constructed while the system runs. We can constrain that graph, and I’ll argue below that we must, but we can no longer assume every path is known before the request arrives.
I learned this the slow way, building agentic services that are either customer-facing or reusable by other product teams. Every design taught me the same lesson from a different angle. The hard part was never the model. It was working out what “well-architected” even means when the system in front of us reasons instead of merely responding.
This isn’t just my experience
For a while we treated the new edge cases as best-effort work, because the agents were proofs of concept and nobody had promised anything. That stopped being tenable as real products started shipping agentic flows.
Take these three published guidelines for example. AWS put out the Well-Architected Agentic AI Lens on June 10, 2026, stating plainly that agentic systems introduce architectural dimensions existing cloud and GenAI guidance doesn’t address. It doesn’t replace the six pillars; it reinterprets them around five characteristics. Agents reason, meaning one request triggers many inference calls and retrievals. They act autonomously, invoking tools without human instruction at each step. They behave stochastically, so the same input can yield different output across runs. They collaborate with each other and take wildly different direction based on a good number of parameters. And they remember, which brings privacy, integrity, and cost problems stateless REST services never had. Google’s Cloud Architecture Center describes the same shift in nearly identical terms. In December 2025, OWASP published its first Top 10 for agentic Applications, built from incidents observed in production rather than from research prototypes.
Two hyperscalers and a security nonprofit converging on the same diagnosis inside a year is a signal. Things really took a turn and we must adapt.
The cheapest question comes before any pillar
Should this task be agentic at all?
The cheapest way to reduce agentic AI risk is to not create it on a whim and just because we could. We must know why it should be agentic. Google’s guidance is refreshingly blunt on this: agents earn their keep on open-ended problems that need autonomous decisions or dynamic multi-step orchestration. For predictable, repeatable, structured work like executing a fixed workflow, summarizing a document, translating text, or classifying feedback, it explicitly recommends the simpler non-agentic approach as cheaper and more efficient.
That distinction gets lost fast in a gold rush. Every engineer wants to sharpen their AI skills, every team wants to build something real, and they should. But building to learn, building for fun, and building a product a customer relies on are three different games with very different bars. If we are not willing to put in the effort to make the product rigorous enough, better seek an alternative path.
A wildly complex agentic solution I recently built in a hackathon met a comment from a close-friend and respected product leader with “this could be done with SQL”. She was referring to a very specific run with a very specific repeatable outcome (repeatability is critical in healthcare) of the solution that we demoed. Yes, that one specific run in that demo would be possible with a wildly complex set of SQLs if we knew that context data and call sequence in advance. Enabling the various scenarios and outcomes may end up causing a combinatorial explosion. Therefore, that hackathon solution needed to determine its path autonomously based on available data, versioned guidance/prompt, and the clinical context in a large hospital setting with multiple regional hospitals. That solution had to be agentic. But that question should be asked, do we really need agentic AI or simply LLM, or non-AI software, or just SQL.
At the same time, that solution needed to drive repeatable outcome. This part was harder, almost twice as much work as everything else combined, to make the agents make repeatable decisions or call out where the context is a grey area, and for good reason. Confusion is not acceptable in healthcare. A patient’s care experience cannot be decided on “I am not sure”. But the context does not help at all. The data that arrives is mostly incomplete. It carries terms that only clinicians at one particular site would recognize. Despite the interoperable data standards including HL7 and FHIR, each hospitals defines custom segments (in HL7) and custom resources (in FHIR). In all that madness, the software can only wonder — stuck in a grey area. However, human clinicians will confidently choose care decisions differently and all can be defensible. But the same is not acceptable for agentic AI. All outcomes must be testable and validated beforehand. The repeatability requirement in healthcare is real and it is not going to relax.
Now keep all the above in mind and rethink, is agentic AI the right tech for the solution?
Operational excellence: a healthy agent can still be a wrong agent
In traditional ops reviews, we ask what is availability (4 9s, 5 9s or better), distribution of 5xx/4xx, and p90/p99/tm90 of every API call’s latency; what are the patterns and how they are trending. Agentic AI ops has to also ask whether its behavior is still intact — inside the boundary we drew. A service can post four nines while its decision quality quietly rots. A prompt tweak that we considered improvement, an edited tool description, a model migration, a changed memory policy: any of these can move behavior without tripping a single availability alarm.
Two practices make that tractable.
Put prompts under the same lifecycle discipline as code. Versioned, deployed through a pipeline, evaluated against a golden dataset before promotion, rolled back the way we’d roll back a bad build. The AWS lens formalizes this as prompt lifecycle management alongside behavioral monitoring, including LLM-as-judge evaluation for outputs with no single correct answer.
Then instrument the reasoning, not just the requests. The OpenTelemetry GenAI semantic conventions define standard gen_ai.* span attributes for model calls, agent runs, and tool executions, so every step in a reasoning chain becomes a child span in one trace tree. They're still pre-stable and attribute names shift between versions, so pin the version we build against. Adopt them anyway. When an agent misbehaves we want to answer "why" from a trace in minutes rather than reconstruct it from logs. A team running uninstrumented agents is debugging a distributed system blindfolded. Having full governance and visibility into agents and their thought process is critical.
Additionally, keep a human in the loop review where it counts. In most healthcare workflow, complex reasoning is not finalized just because the agent recommended it. The agent proposes, a clinician confirms, and the system records the inputs, outputs, context, available tools, reasoning or thought process.
Finally, run a canary that executes known test cases and reports whether the agent is still making the correct decisions.
Security: the agent is a new type of principal
Traditional security reviews reason about users, services, networks, and data boundaries. An agent adds a principal unlike any of those: one that interprets untrusted natural language and can be talked into using entirely legitimate permissions for illegitimate ends. That’s what jailbreaking looks like once the model holds tools.
Watch specifically for what Simon Willison named the lethal trifecta: an agent with access to private data, exposure to untrusted content, and the ability to communicate externally. Any two are survivable. All three together mean an attacker who can get text in front of our agent can get our data out of it. Design the solution in a way that this combination cannot occur.
The OWASP Agentic Top 10 gives the rest a vocabulary worth adopting in reviews. Agent Goal Hijack, where an attacker steers what the agent is trying to accomplish. Tool Misuse, where legitimate tools get driven to unintended ends. Identity and Privilege Abuse. Insecure Inter-Agent Communication. The framing I’ve found most useful is OWASP’s least agency, the natural successor to least privilege. The question is no longer only what the agent can access. It’s how much freedom the agent has to act on that access without checking back. Autonomy should be earned, never defaulted.
In practice that means every production agent gets its own identity and a narrowly scoped authorization boundary. One agent should not read the data, recommend the action, execute it, and approve it. And prompt injection belongs on the operational threat list, not the content-quality list, because an agent holding tools turns injection into a path to real-world action.
Mind the plumbing too. GitGuardian’s State of Secrets Sprawl 2026 reports 24,008 unique secrets exposed in MCP-related configuration files on public GitHub during 2025, more than 2,100 of them still valid when detected. Quickstart docs that hardcode API keys into config files deserve a share of the blame. Adopting MCP without securing its configuration is adopting the risk and skipping the benefit.
The question I keep in my head through every security review: what is the blast radius if this agent does what it is technically authorized to do, but for the wrong reason?
Reliability: available, responsive, and wrong
A conventional application is either up or it isn’t. An agent can be up, fast, and wrong. It can hallucinate a hospital’s bed count, misread an acuity signal, repeat an action after a retry, or keep reasoning long past the point where it should have escalated. None of that throws a 500.
Reliability therefore has to cover behavioral failure, not only infrastructure failure. Below four mechanics carry most of the weight in my opinion.
Decompose workflows into atomic, bounded tasks with explicit completion criteria, so a failure stays contained to one step instead of corrupting a whole run. Make every external action idempotent; an idempotency key on the patient bed admission data can make the difference between a retried step and a double-booked hospital bed. Define stop conditions explicitly, including maximum iterations, wall-clock budgets, and confidence thresholds below which the agent hands off to a human or falls back to a deterministic path. Then tier oversight to the risk and reversibility of the action. Automate low-stake, frequent, clear, and repeatable decisions. Gate high-stakes and potentially unclear (as in a grey area) decisions with human in the loop (HITL).
One more game-changing strategy. Wherever we can, reshape the agentic decision into a classification problem rather than an open-ended task. I’m still surprised how often experienced engineers hand an agent a genuinely open-ended ask and then express disappointment as the outcome does not match expectations.
An equally common mistake is to put a ton of great guidance in prompts that are not needed in that context. All we get at the end is just weakened attention head in the transformer model and poor relevance in the output.
And finally, a silly mistake I still see very commonly is writing prompt with certain data in mind without making sure the associated agents or tools will invariably provide that data. When that assumption does not check out, we have a hard time debugging.
Overall, there are plenty of ways an agent can face a challenge in making autonomous decisions like the above. If the agent still makes a decision and carries that out, bad things can happen. A reliable agent isn’t one that always produces an answer or executes an action. It is the one that knows when to execute a task and when it should hand over control to its human.
Performance efficiency: reasoning latency can bite bad
Compute, caching, and network calls still matter, as much as before. But the dominant latency in an agentic workflow is usually the reasoning path itself: how many model calls, how large the prompts are, how many tool calls round-trips, how much context retrieved, how many agents involved, how many handoffs between them.
Three levers do most of the work.
Route models by step, not by system. Not every step needs our largest model. Classification, extraction, and routing can often be run fine with a small, fast LLM, with the expensive models reserved for the actual judgment call. That should be an architectural decision from the get-go, not an afterthought.
Choose agent topology deliberately. Google’s design-pattern guidance catalogs the trade: sequential and parallel workflows that need no model orchestration at all, coordinator and hierarchical patterns that buy adaptability with extra model calls, swarms that buy creative synthesis at the highest cost and latency of the lot. Multi-agent is a trade-off, not an upgrade. Start with one agent and split only when tool count or task complexity measurably degrades it.
Reduce the tool bloat problem, which is Google’s term for stuffing an agent’s context with too many tool definitions or tools with sprawling parameter lists. The documented effects are lower tool-selection accuracy plus hidden extra reasoning loops, so we pay for worse answers twice. Use relevance-based search and neuro-symbolic filters to restrict what goes into your AI prompts.
Cost optimization: simple Agentic answers can still incur crazy bills
Agentic cost doesn’t map to instance-hours. A single request can quietly generate a dozen model calls, several tool executions, memory reads and writes, and coordination traffic between collaborating agents. The user sees one answer. The bill sees the whole trajectory.
So attribute cost at the workflow level, per business outcome rather than per endpoint. The OpenTelemetry, LangFuse, and similar can be used to trace token count and dollars spent per request, session, and time slot. We can easily detect and debug costly and slow sessions with the why. We should set hard limits on both to avoid silent leak and create alarms that trips if the hard limit is reached.
Basically, budget the conversation, not just the compute.
Sustainability: the waste that matters is organizational
The environmental reading still holds. A wasteful agent burns inference on redundant retrievals, needless tool calls, and elaborate multi-agent topologies for jobs one agent could finish. I still get surprised when I see prompts like “...call ALL available tools and use ALL available data…”
Siloed development practice is a reality in many companies. But if every team independently rebuilds orchestration, memory, tracing, evaluation, and guardrails, we end up with a fragmented agent platform that costs too much to run and is nearly impossible to govern. Build modular, reusable foundations early, so the agents can be built faster with less tech debt and better visibility and control.
Seven questions when we review an agentic design
We don’t need a hundred-item checklist to start. These questions expose most of the risk:
- Why does this task need an agent rather than a deterministic workflow? Do we need simple LLM? Could we use non-AI software to solve the same problem? What are the trade-offs?
- What can the agent read, and what can it execute or approve? What is its blast radius for each category?
- What data permissions do the agents have? Can they access database through other services/APIs or directly? Are those data internal or confidential? Do they contain PHI/PII? Can they access the Internet?
- How are its reasoning loops, tool calls, latency, and spend bounded? What does the worst-case execution look like? Where should we put a limit?
- How are prompts, tools, models, and memory policies versioned and evaluated before release? How do we govern the agents, tools, integrations? How do we run weekly bar-raising reviews?
- Where does a human sit before an action becomes irreversible? Who is that user persona? What level of qualifications or skills do they come with? Do they prefer more false positives or more false negatives? What is the right tradeoff and risk appetite?
- Can we reconstruct the evidence, tool calls, state changes, and approvals behind any consequential action? What is covered in dashboards, metrics, alarms, logs, and traces?
Weak answers to these are not implementation details to defer until after the pilot. They’re architectural risks. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, and my read is that answers to these questions may save the ones that are truly differentiating but are going to die because of design risks.
Conclusion
WAF is a great tool to bar raise design and can be applied to agentic solutions with this small addendum. In healthcare, the correctness of a care decision is often more important than availability of a non-medical device-software. However, there are a lot of frequent, clear decisions that we can automate safely using agentic AI. And there are a lot of high-risk care decisions that must be approved by a human clinician. Here agentic AI can be used to make recommendations to assist rather than to make decisions to automate. Either way, build what your customers appreciate and need. And build trust over anything else. And for that, take these few high-impact questions with WAF to whatever your teams are building this quarter, before scale makes the gaps expensive.
References
- AWS Well-Architected Framework — aws.amazon.com/architecture/well-architected
- AWS Well-Architected Agentic AI Lens, published June 10, 2026 — docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens
- OWASP Top 10 for Agentic Applications 2026, published December 9, 2025 — genai.owasp.org/initiatives/agentic-security-initiative
- Simon Willison, “The lethal trifecta for AI agents,” June 16, 2025 — simonwillison.net
- GitGuardian, “The State of Secrets Sprawl 2026” (5th edition; MCP configuration findings cover public GitHub, 2025) — gitguardian.com/state-of-secrets-sprawl-report-2026
- OpenTelemetry GenAI Semantic Conventions (pre-stable; agent, tool, and model span definitions) — opentelemetry.io/docs/specs/semconv/gen-ai
- Google Cloud, “Choose your agentic AI architecture components,” last reviewed April 21, 2026 — docs.cloud.google.com/architecture/choose-agentic-ai-architecture-components
- Google Cloud, “Choose a design pattern for your agentic AI system,” last reviewed May 28, 2026 — docs.cloud.google.com/architecture/choose-design-pattern-agentic-ai-system
- Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027,” June 25, 2025 — gartner.com
