A customer-support agent loop can retry tools, and a timeout can hide a successful feature flag change. Without idempotency, those retries can produce duplicate writes. That constraint changes the design.
TL;DR: send an explicit desired state instead of an invert command, reuse one operation ID across every retry, and observe attempts separately from committed mutations. Measure latency and model cost at the agent-loop boundary. Count a flag mutation only when state actually changes. This preserves failure evidence without turning retry noise into fake writes.
How should feature flag idempotency stop retries from producing duplicate writes?
The agent asks to enable a rollout cohort. Its backend commits the request, but the response misses the caller's deadline. A retry then arrives. From the client side, the first attempt is ambiguous: it may have failed before the server received it, or succeeded before the response disappeared.
An endpoint shaped as toggle() makes that ambiguity dangerous because repeating the command reverses the result. Use assignment semantics: set this flag to enabled: true for this cohort. HTTP defines PUT as idempotent, but the method name cannot rescue a handler whose business operation still means “invert the current value.”
Dashboards can lie here. Counting every HTTP attempt as a write inflates mutations. Counting only received success responses can hide a commit whose response was lost. Keep four units distinct:
| Signal | Unit | Answers |
|---|---|---|
| Agent loop | One support task | End-to-end latency and model usage |
| Operation | One requested target state | Logical outcome |
| Attempt | One HTTP exchange | Timeout and retry behavior |
| Commit | One durable transition | Actual mutation count |
Noise starts when those units share a counter.
Three attempts can still mean one commit. Keep that sentence beside the dashboard design; it is the fastest test for a misleading chart.
The smallest implementation
Create the operation ID outside the retry loop. Reuse the payload too. The client must label a transport failure unknown, because no response is not proof that no commit happened.
type Assignment = { flag: string; cohort: string; enabled: boolean };
type Outcome = "committed" | "replayed" | "rejected" | "unknown";
interface Telemetry {
attempt(event: {
operationId: string;
attempt: number;
durationMs: number;
outcome: Outcome;
}): void;
}
async function assignFlag(
endpoint: URL,
value: Assignment,
telemetry: Telemetry,
maxAttempts = 3,
): Promise<void> {
const operationId = crypto.randomUUID();
for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
const startedAt = performance.now();
let outcome: Outcome = "unknown";
try {
const response = await fetch(endpoint, {
method: "PUT",
headers: {
"content-type": "application/json",
"x-operation-id": operationId,
},
body: JSON.stringify(value),
});
if (response.ok) {
const result = (await response.json()) as { replayed: boolean };
outcome = result.replayed ? "replayed" : "committed";
return;
}
if (response.status >= 400 && response.status < 500) {
outcome = "rejected";
throw new Error(`Assignment rejected: ${response.status}`);
}
} catch (error) {
if (outcome === "rejected" || attempt === maxAttempts) throw error;
} finally {
telemetry.attempt({
operationId,
attempt,
durationMs: performance.now() - startedAt,
outcome,
});
}
}
}
The backend must claim the operation ID and change state in one transaction. Store the result beside that ID. A repeated request returns the stored result and reports a replay; it does not run the mutation again. A unique constraint closes the race between concurrent retries, while the losing transaction reads the winner's stored result. The trade-off is extra transactional storage and another cleanup policy. For a read-only tool call, or a naturally idempotent assignment where duplicate execution has no side effects, a stored operation ledger may add more machinery than signal. For a mutation that triggers billing, notifications, or another downstream action, skipping the ledger leaves the ambiguous outcome unresolved.
Keep changed separate from replayed. Two different operation IDs may request the same desired state. The second is a valid no-op, not a replay and not a second transition.
Correlation without cardinality damage
Put the operation ID in traces and logs. Do not use it as a metric label. Each value is effectively unique, which creates an unbounded metric dimension and adds query and storage pressure.
Represent the support task as a parent span, then attach tool calls and backend attempts as children. Record bounded fields such as attempt number, outcome, and whether state changed. Logs can carry the trace ID and operation ID for exact lookup. Metrics should aggregate bounded dimensions: outcome, retry bucket, and environment are useful; customer text and arbitrary flag payloads are not.
The concrete story should read: one ticket-resolution loop, one requested cohort assignment, two transport attempts, one commit, one replay. The loop owns total latency and model usage. The operation owns mutation truth. This split shows whether retries delayed support work without pretending they caused extra model usage or extra state changes.
I would benchmark the instrumentation before widening it: events per support task, serialized bytes per event, and query latency for the investigations the team actually performs. Start with three queries, not a catalog of hypothetical dashboards: find all attempts for one operation, compare retry count with loop latency, and reconcile committed transitions with replayed requests. Sampling can preserve broad latency distributions, but it is a poor substitute for the durable operation record when a rare mutation is disputed. This is a deliberate signal-quality trade-off. Exact operation history costs storage; aggressive sampling costs the evidence needed to settle the hardest cases.
What changes at scale
Give idempotency records an explicit retention rule. The record cannot expire before the maximum credible retry window, including delayed queue delivery. There is no universal duration; derive it from the delivery policy, then test requests immediately before and after expiry.
Export analytical events after the operational transaction commits. ClickHouse is an analytical database and can store high-volume attempt data, but the analytics copy should not decide whether a mutation already happened. That answer belongs to the transactional store.
Then test the uncertain boundary. Drop a response after commit. Send the same operation ID concurrently. Retry after the client deadline. The invariant is precise: one operation ID produces at most one committed transition, while every network attempt stays visible.
More fields are not automatically better. Full prompts, customer messages, and flag payloads raise privacy exposure and storage noise. Record identifiers and bounded classifications unless a specific investigation and data policy require content.
The integration is observable enough when an engineer can answer four questions without manually reconciling counters: How many support-agent loops ran? How many logical assignments were requested? How many transitions committed? How many attempts ended with an unknown client-side result?
If one chart claims to answer all four, the data model is too compressed. Separate attempt evidence from mutation truth, correlate both with a stable operation ID, and make the command repeatable. This approach does not fit every endpoint: low-value, side-effect-free calls may be better served by plain retry metrics, while long-running workflows may need a durable state machine instead of a single request ledger. Fewer impressive counters. Better incident evidence.
