A breakout strategy passed all six parameter cells. It was a fill assumption, not an edge.
I had a breakout strategy on 1-minute data come in at t = 6.22.
Six parameter cells, all six passing. That is the kind of number that makes you start thinking about position size and funding an account. It is also the kind of number that should make you suspicious, because in ten years of doing this I have almost never seen a real edge survive a clean audit with a t-stat that high.
It was not real. The strategy was fine. The fill assumption was wrong, and it was wrong in exactly the direction that flatters you.
Here is the whole story, because the same bug is probably sitting in a lot of backtesters, including yours.
The bug
My breakout entry looked like this:
if high[j] >= buy_level:
entry = buy_level
buy_level came from the previous session's high. The logic reads fine: price trades up through my level, I get filled at my level. That is what a limit order does.
The problem is the gap. When the new session opens with a gap straight through buy_level, bar j satisfies high[j] >= buy_level on its very first tick — at the open. The open is already above my trigger, and I record a fill at buy_level, a price that never traded in that session.
I was gifting myself the gap. Systematically, on every gap-through day, in the direction that made the equity curve look better.
The X is the recorded fill. The dotted line is that bar's low. The fill sits 1.48 below the lowest price of the bar — a price that, by construction, no one could have transacted at. The backtester did not complain, because nothing in the code knows what a plausible fill looks like.
The fix
One line:
entry = max(buy_level, open[j]) # the worse of the two, for a long
If the bar opened above my trigger, I get the open — the price I could actually have paid. If it opened below, I get my trigger, as intended. Same for shorts with min(...).
That is the entire change. Not a filter, not a parameter, not a new feature.
Same trades, same period, same parameter grid. 6.22 → 1.04. Passing cells: 6 of 6 → 0 of 6.
The entire edge lived in the fill.
Why this is hard to catch
Three reasons, and they compound.
It is plausible. entry = buy_level reads like correct code. It is the natural way to express "I got filled at my level". Nothing about it looks like a mistake, so code review slides right over it.
It inflates results in the direction you want. There is no error, no crash, no unusual log line. The only symptom is a nicer equity curve, which is precisely the thing you were hoping for. Your brain is not going to flag good news as a bug.
It hides in the gap. If you trade a market that does not gap, you may never see it. It only appears on days where the open jumps past the trigger — and on those days it silently substitutes a favourable price for the one you would have got.
How often it actually bites
I measured the share of entries where the recorded fill fell outside the bar's [low, high]:
The pattern is exactly what you would expect: the more a market gaps overnight, the worse it gets. Index futures on a session-based breakout are the worst case — roughly one entry in six.
And note what this does to the interpretation of a backtest. On NQ, 17.4% of entries were priced at a level the market never traded. If your strategy's edge is on the order of a few basis points per trade, and one entry in six is being handed a free gap, then the measured edge is mostly the gap.
How I found it
Not by reading the code. I had read that code a dozen times.
I found it by looking at when the largest 1-minute moves happened. I ran a data-quality pass that printed the twelve biggest 1-minute jumps with their timestamps, and all twelve landed between 18:00 and 18:05 New York time.
That clustering was the tell. Twelve enormous 1-minute moves, all in a five-minute window, every time — those are not market moves. That is the session boundary. The "jumps" were cross-session gaps being recorded inside a session-labelled dataset.
Once I was looking at gaps specifically, the fill assumption fell apart in about ten minutes.
The general lesson here is worth more than the bug: when you audit a backtest, look at the time distribution of anomalies, not just their size. A 2% one-minute move at a random time is a market event. A 2% one-minute move that always happens at 18:00 is a data-structure artefact, and it is telling you that your data is not shaped the way your code assumes.
What this generalises to
Fill assumptions are where backtests lie most often, because they are the one part of the simulation that has no external check. Your signal logic produces a number you can inspect. Your fill price is just... whatever you wrote.
Three specific forms to look for:
- Gap-through fills — the one above. Trigger levels set from prior session data, filled at the trigger regardless of the open.
-
Look-ahead on exits — if
execute()is only called on bars where a signal fired, then exits are only checked on those bars. I had this one too: fixing it moved a different strategy from −5.7% to +17.9%, and inflated the win rate by 6–8 points. The original reported a 74.1% win rate; most of that was the bug. -
Cost assumptions borrowed from another asset class — two futures strategies of mine were costed at 5bp per side. Actual round-trip cost was closer to 0.30bp (NQ) and 0.55bp (MGC). That is an 18–33× overstatement. Correcting it moved one of them from −94.0% to −45.5%. Rule of thumb: if
cost ÷ per-trade riskexceeds about 5%, suspect the cost model before you reject the strategy.
Three checks you can run on your own log
You do not need my code for these; you need your trade log and your price bars.
1. Is every fill inside its bar?
For each trade, find the bar containing the entry timestamp and check
low <= entry_price <= high. Anything outside never traded. This single check catches the gap-fill bug directly, and it is the highest-yield check I know.
2. Is the fill better than the bar's open?
For breakout entries, if a long fill is below the open, either a limit order filled later in the bar (fine, but then model it that way) or you have gifted yourself the gap (not fine). This one has false positives — limit orders legitimately fill better — so treat it as a prompt to look, not a verdict.
3. Do your anomalies cluster in time?
Print your largest N moves with timestamps and histogram the hour. If the biggest moves all sit at a fixed time of day, you are looking at session boundaries, and your session handling is probably wrong.
The tool
I turned the checks I now run into a small script, because doing them by hand every time is how you stop doing them. It is one file, standard library only, no pandas:
python backtest_lint.py trades.csv --prices prices.csv
That image is real output from the tool on a demo log that has five bugs deliberately planted in it. It scores 1/6 and names each one with the row number and the actual numbers. The same repo also ships a clean control that scores 6/6 — a checker that always says "fail" would be worthless, so both directions are testable before you trust it.
Code: https://github.com/wei498766-ai/backtest-lie-detector — MIT, no dependencies.
The uncomfortable part
The uncomfortable part is not that I had a bug. Everyone has bugs.
The uncomfortable part is that the bug made the result better, and the result looked like a discovery. I did not find this because I was careless. I found it because I got suspicious of a number that was too good, and then went looking for how it could be too good.
That is not a scalable process. It depends on being suspicious at the right moment, about the right number — and the numbers you most want to believe are exactly the ones you are least likely to question.
So the process has to be external. Cheap, mechanical checks that run whether or not you are in the mood to doubt yourself.
That is the whole argument for this kind of tooling. Not that it makes you a better trader, but that it does not care what you want to be true.
If you have hit something in this family — particularly anything where the backtester's assumption is invisible in the output — I would like to hear about it. The ones that do not throw errors are the ones worth collecting.
If you would rather have this as a download than a repo: the same tool, plus the
demo trade logs (one with five bugs planted, one clean), is *$0** on Gumroad —
pay what you want, and $0 is a valid answer: https://weiwave23.gumroad.com/l/backtest-lie-detector*
