Our tool gateway says max_retries=2. It calls your tool exactly once.

The short answer

How many times does one tool call actually execute in our agent runtime?

Exactly once — while the policy gateway's constructor says max_retries: int = 2.

Those two facts do not contradict each other. Three independent reasons stack up:

1. That 2 is unreachable in the shipped wiring. Every construction site passes a trace_writer, so every call gets an idempotency key, and the gateway clamps the attempt count to 1 whenever it sees one.
2. Even if it were reachable, it does not mean "retry twice." The shared helper hands the number to tenacity's stop_after_attempt, which counts attempts.
3. Even if a retry happened, almost nothing would trigger it. All three tool adapters swallow their exceptions into ToolResponse(success=False), and tenacity only retries on exceptions that are actually raised.

What really keeps a side effect from happening twice is none of the above. It is the
layer below: a durable tool-call ledger with a lease and a column called
outbound_started_at.

For contrast, the other boundary in the same repository — outbound events — defaults
to at-least-once, up to 64 attempts.

This is not a bug list. Most of it is deliberate. The naming and the documentation
are the parts that genuinely do not line up, and they are all in the second-to-last
section.

1. This started as a reply to a comment

On 2026-09-07, under our post about agent frameworks and agent runtimes being
different layers, reidmarlow (Reid Marlow) left a comment. Along with agreeing that
splitting resolved from redacted parameters at the port layer is the cleanest part of
the architecture, he added an argument we had not made ourselves:

Pushing the retry clamp down into the policy gateway also solves a second problem —
when a downstream service cannot honor an idempotency key, a timeout retry fires the
external side effect twice.

Good argument. It is also why this post exists: before replying, I wanted to check
whether our code actually works the way he read it.

It does not. The direction is inverted.

Our gateway does not say "the downstream has no idempotency key, so retry less." It
says "this call carries an idempotency key, so try it exactly once." The path
without a key is the one that keeps the max_retries default.

His conclusion still holds here — side effects do not fire twice — but it holds for a
reason one layer further down. Explaining that requires counting all three layers.

2. Layer one: the ternary that decides everything

Before the gateway reaches an adapter it resolves secrets, claims a ledger row, checks
egress policy, and checks rate limits and quota. Then it reaches this:

response = await run_with_timeout_retry(
    _invoke,
    timeout_seconds=timeout_seconds,
    # Durable Agent calls are at-most-once at this boundary.
    # Not every downstream adapter can honor an idempotency key.
    max_retries=1 if kwargs.get("idempotency_key") else self.max_retries,
    timeout_factory=lambda: TimeoutError(
        f"Tool invocation timed out after {timeout_seconds} seconds",
        {"timeout_seconds": timeout_seconds, "tool_ref": tool_ref},
    ),
    wait_min=1,
    wait_max=5,
)

(server/app/kernel/ports/tools/policy.py:349–361)

The two comment lines state the intent plainly: durable agent calls are at-most-once
at this boundary, because not every downstream adapter honors an idempotency key.

Which means the idempotency_key in that ternary is not a capability signal — it is
a path marker. It marks "this call is the kind that has a ledger behind it." That is
a different thing from what Reid read it as, and the difference is exactly what decides
which way the clamp points.

3. max_retries counts attempts, not retries

Five ports — tools, storage, vector, secrets, plugins — share one helper:

async def run_with_timeout_retry(
    operation, *, timeout_seconds, max_retries, timeout_factory,
    wait_multiplier=1, wait_min=1, wait_max=10,
):
    @retry(
        stop=stop_after_attempt(max_retries),
        wait=wait_exponential(multiplier=wait_multiplier, min=wait_min, max=wait_max),
    )
    async def _with_retry():
        return await operation()

    try:
        return await asyncio.wait_for(_with_retry(), timeout=timeout_seconds)
    except TimeoutError:
        raise timeout_factory()

(server/app/kernel/ports/common/policy.py:52–74)

stop_after_attempt counts attempts. stop_after_attempt(2) means two calls total,
i.e. one retry. The parameter is named max_retries, its docstring says "Maximum
retries", and the number it actually carries is one less than its name suggests.

I am not inferring this. Our own unit test pins it:

result = await run_with_timeout_retry(
    operation, timeout_seconds=5, max_retries=2, ...
)
assert result == "ok"
assert attempts == 2

(server/tests/unit/test_port_policy_common.py:32–52)

The direct consequence: max_retries=1 means "do not retry." And the repository
passes a literal 1 in six places:

Every one of those is read-only or naturally idempotent — precisely the category that
should be retried. I do not think this was intentional. I think someone read 1 as
"retry once."

For reference: storage defaults to max_retries: int = 3 (three attempts), vector to
2, tools to 2. All three docstrings say "Maximum retries."

4. Which makes the shipped answer 1

Back to the ternary. When does kwargs contain an idempotency_key?

The gateway puts one there itself, as long as it was constructed with a trace_writer:

tool_call_id = str(kwargs.get("tool_call_id") or f"call_{generate_ulid()}")
idempotency_key = str(
    kwargs.get("idempotency_key") or f"tool:{run_id}:{tool_call_id}"
)

(policy.py:261–264, written back into kwargs at :298–303)

So: can a shipped tool port ever be built without a trace_writer?

There is exactly one place that constructs this gateway
(app/wiring/container.py:487), and all four of its call sites pass a TraceWriter
constructed on the spot:

• app/wiring/services.py:414–416 (agent application service)
• app/api/v1/agent/dependencies.py:47–49 (HTTP dependency)
• app/modules/workflow/runtime/engine.py:558–561, :606, :693–696 (workflow engine)

No production path leaves trace_writer as None. The self.max_retries branch
is unreachable in the shipped wiring, and that default of 2 is dead code today.

It is also deliberate. The test says so in its name:

async def test_tool_policy_does_not_retry_a_durable_agent_invocation(request_ctx):
    ...
    policy = ToolPolicyGateway(gateway=SideEffectingTool(), ctx=request_ctx,
                               max_retries=2, enable_egress_check=False)
    with pytest.raises(RetryError):
        await policy.invoke(..., idempotency_key="agent-tool:run_side_effect_once:call_once")
    assert attempts == 1

(server/tests/unit/test_port_policy_enforcement.py:375–399)

It passes max_retries=2 explicitly and asserts attempts == 1. This is a tested
at-most-once boundary, not an oversight.

One more thing worth saying out loud: this gateway's timeout_seconds and
max_retries have no configuration entry at all. The wiring does not pass them and
settings.py contains no setting starting with tool_. Changing either means changing
code. (Rate limit and quota do have an entry — they come from the request context, as
ctx.tool_rate_limit_per_minute and ctx.tool_daily_quota.)

5. And even that 2 would rarely fire

tenacity's @retry only retries on exceptions that get raised. None of our three
tool adapters raise:

# adapters/tools/http.py:97–101
except httpx.HTTPError as e:
    return ToolResponse(result=None, success=False, error=str(e))

# adapters/tools/function.py:41–42
except Exception as exc:
    return ToolResponse(result=None, success=False, error=str(exc))

# adapters/tools/mcp.py:323–325
except Exception as exc:
    logger.warning("MCP tool invocation failed for %s: %s", tool_ref, type(exc).__name__)
    return ToolResponse(result=None, success=False, error=str(exc))

httpx.HTTPError is the base class for that library's errors: connection failures,
read timeouts, and everything raise_for_status() raises for 4xx and 5xx.
All of it becomes a return value rather than an exception.

So an HTTP 503, a connection timeout, or an MCP server falling over — the textbook
"worth one retry" transients — are never retried on this path, not even on the
branch that allows retries.

What does get retried? Whatever the tool router raises:

raise ValidationError(f"Tool not registered for tenant/workspace: {tool_ref}")   # router.py:364
raise ValidationError(f"HTTP tool '{tool_ref}' missing url (tool_spec.http.url)") # router.py:454
raise ValidationError(f"Function tool '{tool_ref}' missing entrypoint")          # router.py:512
raise ForbiddenError(...)                                                        # router.py:301

Configuration errors and permission errors — the ones no amount of retrying will
fix. With a wait_exponential(min=1, max=5) sleep in between.

This is not inevitable. The LLM port next door has an _is_retryable predicate and a
test that asserts validation errors are not retried
(test_validation_errors_are_not_retried, test_port_policy_enforcement.py:236–250,
asserting await_count == 1). The tool port has no such predicate.

6. The timeout is a budget for the whole sequence

Look at the last line of that helper again:

return await asyncio.wait_for(_with_retry(), timeout=timeout_seconds)

wait_for wraps the entire retry sequence, not a single attempt. So the tool
port's default 30 seconds is the budget for every attempt plus the backoff between
them.

Meaning: if the first attempt burns the full 30 seconds, a second attempt never
happens — the timeout lands first and gets converted into our own TimeoutError by
timeout_factory(). The already-hard-to-reach retry path only has room when the first
attempt fails fast.

The parameter is called timeout_seconds, and the tool gateway's docstring calls it
"Request timeout" (policy.py:75). It is not a request timeout. It is a total
timeout.

For contrast, the LLM port's streaming path wraps each attempt in its own wait_for
(ports/llm/policy.py:932–937). Same repository, two shapes.

7. Failures leave the gateway wearing a different type

tenacity's @retry defaults to reraise=False: once the stop condition is met and the
last attempt still failed, it raises RetryError with the original exception inside.

This is independent of the retry count. Even with stop_after_attempt(1), a failed
first attempt comes out wrapped. So in the shipped wiring — which always takes the
at-most-once branch — every failure leaving the tool gateway is a RetryError, not
a ValidationError, not a ForbiddenError.

There is an unwrap_retry_error helper for exactly this
(ports/common/policy.py:38–49), but it is only used by error_details to write trace
detail. The copy that gets unwrapped is the one for the books; the exception actually
propagating upward is not swapped back.

Two unit tests have already frozen this into assertions:
test_port_policy_enforcement.py:391 and test_tool_secret_injection.py:215, both
with pytest.raises(RetryError).

The impact is concrete: any caller upstream writing except ValidationError: to turn
"tool not registered" into a 400 will not catch it on this path.

8. Layer two: tool_call_id is the identity, idempotency_key is an assertion

Before touching an adapter, the gateway claims a row in run_step_tool_calls.

That table carries three unique constraints
(kernel/runtime/db/models/runs.py:185–203); two of them matter here:

Only the first is used for lookup. _call_statement finds an existing row by
tenant + workspace + run_id + tool_call_id (runtime/runs/tool_calls.py:172–180).

The idempotency_key's role here is an assertion, not a key: once a row is found,
a mismatching tool_ref or idempotency_key raises
ConflictError("Tool call identity was reused with different input")
(tool_calls.py:215–219). The arguments have to match too — compared through
canonical_request_hash.

The default key looks like tool:{run_id}:{tool_call_id}, so on the default path the
two constraints never fight.

One more detail worth noting: what lands in the ledger is the redacted payload
(policy.py:283 passes redacted_parameters); a secret reference keeps only its
secret_id, never the plaintext.

9. How at-most-once actually reaches the disk

The real guarantee lives in three columns: status, lease_expires_at, and
outbound_started_at.

The order is:

1. claim() writes a row with status="claimed", attempt_count=1, and a lease owned by this worker for max(60, ceil(timeout) + 10) seconds (policy.py:274).
2. mark_running() stamps outbound_started_at before crossing the boundary (tool_calls.py:473–495). That single write is the pivot of the whole mechanism.
3. The adapter is called. Each attempt starts by renewing the lease (policy.py:328–329).
4. complete() writes the terminal state: status = "succeeded" if response.success else "failed" (tool_calls.py:543).

If the process dies during step 3, the next worker to claim finds a row with an expired
lease, and splits on one column:

• outbound_started_at is null — the request never left. Safe. Re-claim it and bump attempt_count (tool_calls.py:322–332).
• outbound_started_at is set — we went out and do not know what happened. Do not re-run. Mark the row in_doubt, mark the step paused, raise ConflictError("Tool call outcome is in doubt") (tool_calls.py:297–320).

This is where "the side effect does not fire twice" actually comes from. It does
not depend on downstream idempotency support, and it does not depend on how many times
the gateway retries. It depends on the fact that going out the door is itself written
down.

Reid's conclusion holds here because of this layer, not because of that ternary.

10. Replay and re-run are one boolean apart

When a row that already reached a terminal state is claimed again, the default is
replay:

if existing.status in {"succeeded", "failed"}:
    ...
    return ToolExecutionClaim(
        record=existing, run_step=step, replayed=True,
        cached_response=ToolResponse(
            result=payload.get("result"),
            success=existing.status == "succeeded",
            error=existing.error_message,
            metadata={..., "idempotent_replay": True},
        ),
    )

(tool_calls.py:270–289)

Note that {"succeeded", "failed"} contains both: a failure is replayed verbatim.
The gateway sees replayed=True and returns the cached response without ever reaching
an adapter (policy.py:291–296).

Making it actually re-run requires passing retry_failed=True, which clears the
result, bumps attempt_count, marks the step retrying, and starts over
(tool_calls.py:249–269).

And that boolean differs between the two execution paths:

Both sides have decent comments (the workflow one says the identity is attempt-stable,
so a node retry lands on the previous attempt's record: replay it when it succeeded,
re-execute it when it failed). The divergence itself is written down nowhere.

11. Layer three: workflow nodes, and one name with three meanings

The workflow engine carries a retry loop of its own:

retry_policy = node.get("retry_policy") or policy.get("default_retry_policy") or {}
max_retries = int(retry_policy.get("max_retries", 0) or 0)
...
attempts += 1
if attempts > max_retries:
    final_error_recorded = True
    raise

(modules/workflow/runtime/executor.py:427–428 and :483–485)

Here max_retries really does count retries: the default 0 means no retry, and 2
means at most three executions. Each retry also creates a new step, suffixed
_retry{attempt} (executor.py:380).

So the same name means three different things:

The LLM line is at ports/llm/policy.py:921, and its count is pinned by a test as
well: test_max_retries_counts_additional_attempts asserts
port.chat.await_count == 3 for max_retries=2
(test_port_policy_enforcement.py:219–232).

In other words: the tool port is the only one of the three that is off by one, and
it happens to be the only one that produces external side effects. On the bright side,
that is the safe direction to be wrong in.

12. How many times the downstream saw it, the ledger counted it, and the bill counted it

These three numbers are not necessarily equal.

The idempotency key does reach the downstream — on four methods:

idempotency_key = kwargs.get("idempotency_key")
if idempotency_key and method.upper() in {"POST", "PUT", "PATCH", "DELETE"}:
    headers.setdefault("Idempotency-Key", str(idempotency_key))

(adapters/tools/http.py:34–36)

GET does not carry it. Reasonable — but "we pass the idempotency key downstream" needs
an "except on GET" attached to it.

The MCP adapter retries once on its own. When an MCP server answers 401 or 403 with
a WWW-Authenticate challenge, the adapter re-derives a token from the scopes in that
challenge and calls again (adapters/tools/mcp.py:77–94). A test asserts it:
assert len(factory.calls) == 2 (tests/unit/test_mcp_official_sdk_adapter.py:272–312).

That retry is invisible upward: the gateway counts one invocation, the ledger's
attempt_count stays at 1, and the MCP server saw two sessions.

Rate limit and quota are checked once, before any retry. Both check_rate_limit
calls sit at policy.py:311–325; the retry happens at :349. So on that (currently
unreachable) non-durable branch, two downstream requests would consume one unit of
quota.

Billing counts once too: record_cost(..., billed_quantity=1, ..., request_count=1)
(policy.py:419–430), one entry per invoke.

The platform has already thought carefully about this exact class of problem in one
other place, and that docstring deserves quoting in full:

llm_image_max_retries: int = 0
"""
Retry ceiling for image generation, capping the per-route retry budget.

Image generation is billed per generated image and is not idempotent: a
platform-side timeout does not cancel the provider-side generation, so a
retry can bill the workspace again while the platform records a single
usage fact. Zero keeps recorded cost aligned with provider charges.
"""

(settings.py:240–247)

Every word of that transfers to tool calls. Tool calls have no equivalent note and no
equivalent knob.

13. The other boundary defaults the opposite way

Outbound events in the same repository chose the mirror image:

max_dispatch_attempts: int = 64
...
next_attempt = int(row_fresh.attempt_count or 0) + 1
if next_attempt >= self.max_dispatch_attempts:
    await self.repo.mark_failed(row_id, msg, consumer_name=consumer_name)
else:
    await self.repo.mark_retry(row_id, msg)

(kernel/events/dispatcher.py:50 and :76–88)

At-least-once, up to 64 attempts, with backoff handled by the outbox repository.
Deduplication is the consumer's job: a (consumer_name, event_id) checkpoint table
where the unique constraint absorbs the duplicate
(kernel/events/checkpoint.py; an insert conflict returns False, meaning "already
handled").

The contrast is defensible. Events are ours, replayable, and consumers can be made
idempotent; tool calls belong to someone else and cannot be taken back. Two
boundaries defaulting opposite ways is evidence that somebody thought about it.

One thing does not line up, though. The event side is documented — "provides
at-least-once delivery; consumers remain responsible for idempotent side effects"
(README.md:77). The tool side's at-most-once contract appears nowhere in the
docs. It lives in two comment lines at policy.py:352–353 and in the name of one
test.

14. So was the comment right?

Three sentences:

1. The direction is inverted. We do not clamp because the downstream lacks an idempotency key; we clamp because this call has one, which is our marker for "this one has a ledger."
2. The conclusion holds. Side effects do not fire twice.
3. It holds one layer down. It comes from the table with outbound_started_at, and from that table choosing in_doubt over a re-run when the outcome is unknown.

If you are building something similar, two takeaways from this exercise:

• Decouple "how many attempts" from "is there an idempotency key." The key is a signal about the downstream's capability; the retry budget is your own risk appetite. Binding them together with a ternary means nine readers out of ten read it backwards — and one reader did, in a direction more intuitive than what our code actually does.
• Real duplicate protection lives in a table, not in a counter. A counter only decides how many times you try. Only a row can tell you whether the last attempt made it out the door.

15. Twelve things that do not line up

House rule: whatever I found, I list. Each one gets its impact and a workaround.

① max_retries counts attempts while its name and docstring say retries. Five
ports share the helper; six call sites pass a literal 1 (see the table in section 3),
which means "no retry", and all six are read-only or idempotent operations. Impact:
readers overestimate the system's retry behavior. Workaround: read it as
max_attempts. I plan to open an issue.

② The tool boundary's at-most-once contract only exists in two comment lines and a
test name. The README documents at-least-once for events and says nothing about
tools. Impact: a self-hoster cannot tell from the docs whether their tool call can be
re-sent. I plan to open an issue.

③ Three adapters swallow exceptions into success=False, so the gateway's retry is
dead for all of them — while configuration errors from the router do get retried.
Impact: the transients most worth retrying are not retried, and the config errors least
worth retrying burn 1–5 seconds of backoff. Workaround: everything takes the
at-most-once branch today, so the practical damage is limited to the retry semantics
being nominal.

④ Failures leave the gateway as RetryError with the original type buried inside.
unwrap_retry_error is only applied to trace details, not to the propagating
exception; two unit tests have frozen this as an assertion. Impact: upstream code that
maps exception types to HTTP status codes cannot catch it. I plan to open an issue.

⑤ timeout_seconds wraps the whole retry sequence while its docstring calls it
"Request timeout." Impact: when the first attempt eats the budget, the second never
happens, and the parameter name does not say so. The LLM streaming path is per-attempt
— two shapes in one repository.

⑥ The tool gateway's timeout and attempt count have no configuration entry at all.
The wiring does not pass them and no setting starts with tool_. Impact: self-hosters
have to edit code. I plan to open an issue.

⑦ 🔍 Inference, not observation: the HTTP adapter passes timeout=timeout_s
explicitly to httpx, and that timeout_s is only populated when the tool spec declares
policy.timeout_ms or http.timeout_ms (adapters/tools/router.py:489–502);
otherwise it is None. Per httpx's API, omitting the argument uses the
USE_CLIENT_DEFAULT sentinel, while passing None explicitly means Timeout(None),
i.e. "No timeouts" (httpx/_client.py:352, :370–374; httpx/_config.py:72–84,
read from the library source in our own virtualenv). So a registered tool with no
declared timeout bypasses the client's 30-second timeout and is bounded only by the
gateway's 30-second wait_for. I did not construct this call to prove it, hence
"inference".

⑧ Idempotency-Key is only set on POST/PUT/PATCH/DELETE. Probably correct, but the
sentence "we forward the idempotency key" needs that qualifier.

⑨ 🔍 Inference, not observation: the ledger has two relevant unique constraints, and
claim() only re-queries by run_id + tool_call_id after catching an IntegrityError
(tool_calls.py:384–389). If two different runs supplied the same explicit
idempotency key, the collision would hit the second constraint, the re-query would come
back empty, and a raw IntegrityError would propagate instead of a ConflictError.
The default key embeds the run id, so it cannot collide — which is why I could not
construct this one either.

⑩ The agent path does not pass retry_failed; the workflow path passes True. The
same failed tool call means different things on the two paths: one replays the failure
forever, the other really re-sends. This is plausibly deliberate (an agent loop can
decide to pick a different tool), but it is documented nowhere. I plan to open an
issue.

⑪ The MCP adapter's 401 retry is invisible upward. The downstream sees two
requests while attempt_count stays at 1. Impact: reconciling call volume against the
ledger will not add up. Workaround: that retry only happens on an auth challenge, and
business side effects usually happen after auth, so the exposure is small.

⑫ Quota is decremented once and cost is recorded once, while (on the currently
unreachable branch) the downstream could have been hit twice. The platform already
recognized and handled the same shape for image generation
(llm_image_max_retries: int = 0); the tool path has no equivalent note. Impact: zero
today, because that branch is unreachable — but it returns the moment someone
constructs the gateway without a trace_writer.

Full disclosure

• Nothing was executed for this post. Not one command. Every claim comes from reading source and tests in the soit/ repository at commit abf3dc3, plus the httpx library source in our own virtualenv. I did not stand up an environment to watch a crashed row turn into in_doubt, even though the code path is clear. Items ⑦ and ⑨ above are explicitly marked as inference.
• I only read the community edition. If the Enterprise or Cloud editions wire retries differently, that is out of scope here.
• "Unreachable in the shipped wiring" is a claim about the current wiring, not about every possible caller. The evidence is that the gateway is constructed in one place and all four call sites pass a trace_writer. Construct one yourself without it and that branch comes straight back to life.
• None of the twelve items is a security incident. They are naming, documentation, and layering mismatches — not "somebody got in."
• This post started as someone else's comment. The argument was Reid's, not mine. All I did was go back and check, and find that the direction was reversed.
• Disclosure: I maintain SOIT.

One-sentence takeaway

"How many attempts" and "is there an idempotency key" are two different questions,
and binding them together in a ternary guarantees that readers get it backwards; what
actually keeps a side effect from happening twice is never the counter, it is a row
that remembers whether the request made it out the door.
Our answer happens to be 1 — but it is a 1 stacked out of three independent reasons,
and only one of them was on purpose.

Come and find the holes

The repository is github.com/soit-ai/soit. Every
claim here is checkable:

1. The ternary is at server/app/kernel/ports/tools/policy.py:349–361; read it together with the two comment lines above it.
2. For the counting semantics, run our own test: server/tests/unit/test_port_policy_common.py asserts attempts == 2 for max_retries=2.
3. The lease state machine from section 9 is in server/app/kernel/runtime/runs/tool_calls.py — grep for outbound_started_at and read downward; the in_doubt branch is the pivot of the whole piece.

If I got something wrong — especially if one of the twelve items in section 15 is me
misreading an implementation — please open an issue and say so. I would much rather
hear where it does not line up than be told the design looks clean.