Originally published at rantum.xyz, which is the canonical source.
A revert rate answers one question: how often did a swap fail on-chain? It reads like a clean reliability number, and a leaderboard that ranks venues by it looks like a fair reliability ranking. It is not one, because the number only means the same thing across venues that fail the same way.
This is a short account of a measurement trap that is easy to fall into, hard to see once you have, and directly hostile to a benchmark that claims to be neutral. It is also a record of ClearTrace falling into it, shipping the mistake to production, and then building the fix into the pipeline so the invalid comparison cannot be made again.
The same number, two different measurements
Start with how a swap actually fails, because that is what decides whether its failure is countable.
On a router, the user signs and submits the swap. 1inch, Uniswap, KyberSwap, ParaSwap, and Odos all work this way. If the swap fails, it fails on-chain, ClearTrace sees it, and the revert rate is exactly what it looks like: the odds your transaction does not land.
On a batch-auction venue like CoW Protocol, the user signs an order, not a transaction. A solver wins the batch and submits one settlement transaction for everyone in it. An order that cannot be filled, from a bad price, no solver, or expiry, simply never becomes a transaction. The user's failure is invisible on-chain by construction.
On an intent venue like 1inch Fusion, the shape is the same. The user signs an intent, a third-party resolver fills and submits it, and an unfilled or expired intent leaves no trace on-chain.
So the on-chain revert rate of a batch-auction or intent venue is structurally near zero, and it is near zero regardless of how reliably the venue actually fills. It is not measuring reliability. It is measuring settlement.
The direction of the error is the dangerous part
Measured live on Ethereum over the trailing seven days (September 2026), CoW's revert rate reads 0.53% and 1inch's Limit Order Protocol reads 0.56%, while the routers 1inch and KyberSwap read 1.11% and 1.75%. Sort that single column and the two off-chain venues sit at the top as the most reliable.
That ordering is a category error, and the specific way it is wrong is what makes it corrosive. It flatters exactly the venues a third party can measure least, because their failures never reach the chain to be counted. A benchmark that systematically favors the venues it cannot see is not neutral, whatever its intentions. That is the fastest way to lose the one thing a third-party benchmark is for.
The discriminator is who submits, not where it settles
The tempting shortcut is "off-chain settlement means no reverts." It is wrong in a way that matters. In classic request-for-quote, the market maker signs a quote, but the taker still submits the transaction carrying that signature. An expired quote or a short maker balance therefore does revert on-chain, in the user's own transaction. A venue like that is router-shaped for measurement even though its price came from off-chain.
So the rule is not about where the price originated. It is a single question: does a user's failed swap become a transaction anyone can count? If yes, the venue is a router and its revert rate is comparable. If no, its revert rate is structurally near zero and is not a signal. If the answer is not established, or the venue runs more than one model, it is unclassified and never ranked.
The taxonomy, and the guard that enforces it
ClearTrace encodes those four models as one taxonomy that every consumer imports from a single module, so the rule cannot drift between the dashboard, the API, and the pre-send checks. A short excerpt:
# Execution models
ROUTER, BATCH_AUCTION, INTENT, UNCLASSIFIED = "router", "batch_auction", "intent", "unclassified"
# The ONLY model whose tx-level revert rate measures user reliability, because it
# is the only one where the transaction that fails belongs to the user.
REVERT_COMPARABLE_MODELS = frozenset({ROUTER})
class CrossModelComparison(ValueError):
"""Raised when code tries to rank/compare revert rates across execution models."""
def rank_by_revert(rows, rate_key="revert_rate_pct", project_key="project", strict=True):
"""Rank venues by revert rate, best first, WITHIN ONE EXECUTION MODEL.
Refuses to return a list that mixes models, because such a list ranks nothing."""
rankable, refused = [], []
for r in rows:
(rankable if is_revert_comparable(r.get(project_key)) else refused).append(r)
if strict and refused:
raise CrossModelComparison(
"refusing to rank revert rates across execution models: these venues do "
"not submit the user's transaction, so their on-chain revert rate is not "
"comparable to a router's.")
rankable.sort(key=lambda r: r.get(rate_key))
return rankable
Every published row is stamped with its execution_model and a revert_comparable flag. Those two fields are live on the public API today, so a downstream consumer, whether the ClearTrace dashboard or an outside script, can never read a batch-auction revert rate as reliability without being told otherwise. And a compare page does not print a suppressed head-to-head as a blank. It replaces the number with the reason: a sentence explaining that the two venues settle differently, that the off-chain venue's on-chain revert rate is therefore structurally near zero regardless of how reliably it fills, and that putting it beside a router's would overstate its reliability rather than measure it. The refusal is legible, not silent.
Under-claiming is the default
A venue ClearTrace has not established is unclassified, and unclassified never ranks on revert. Absence of evidence is not evidence. A venue that runs both models is also unclassified: Bebop settles taker-submitted quotes, where a failed fill reverts in the user's own transaction, and solver-submitted intents, where failures never land. A rate over that mixed population answers no question, so ClearTrace does not rank it until the two are split. The same discipline applies to a venue like Tokenlon, whose very low on-chain rate would top any reliability ranking, which is exactly why it needs evidence about who submits rather than an assumption.
Why the guardrail exists
None of this is hypothetical. ClearTrace once shipped the sentence "CoWSwap's revert rate is 0.04% versus ParaSwap's 1.41%" to production, in body copy and in indexed FAQ schema. That is a 35x reliability claim, and it was purely an artifact of settlement: CoW's number was structurally low because unfillable orders never became transactions, not because it executed 35 times more reliably than a router. A flattering wrong number costs a neutral benchmark its credibility exactly as a damaging one does. The taxonomy, the CrossModelComparison guard, and a pre-send claim gate that blocks any cross-model revert claim before it leaves the building are the response to that specific mistake.
What it proves
The rare skill here is not a model or a scraper. It is knowing which comparison is invalid and building the refusal into the pipeline, so a metric that looks universal is only ever computed over the population where it means something. Finding signal in adversarial data includes declining to manufacture signal that is not there, even when the invalid version would make the product look more complete.
For a foundation, a grant program, or a protocol team spending real money against a metric, that is the difference between a dashboard and a source of record: a benchmark whose numbers survive contact with the teams they describe, because the one comparison that would embarrass it is blocked in code rather than caught in review.
The full case study is at rantum.xyz. ClearTrace is at cleartracedata.com.
Andrew Maury is the founder of Rantum, a senior data science and ML studio that turns messy, fragmented, and adversarial data into models, APIs, and products that ship. He has 6+ years in crypto: he previously scaled data at 0x Labs, contributed to Dune's Spellbook, and advised the Uniswap Grants Program.
