I spent a few days auditing a live FX system I built, over 16 years of history. The result was negative in a way I did not expect, and the diagnostic that found it is general enough to be worth writing up — it applies to any parameter search, not just trading.
The setup
17 configurations, 4-hour entry bars, exits simulated on 1-hour bars, 2010-05-28 .. 2026-07-08 (16.1y), 7641 trades. Crucially I fed the unmodified production functions historical CSV instead of a live data feed, so the code path being tested is the code path that runs live.
Bug 1: the simulation charged costs to the wrong place
Platform OHLC bars are BID prices. The backtest placed entry at the bar close, put the take-profit and stop-loss at close ± n·ATR, checked touches against bid highs and lows, then subtracted the spread from the final P&L.
Live, a long position fills at the ASK, the bracket sits relative to that ask, and it closes on the BID. So live needs price to travel one extra spread to reach the target, and reaches the stop one spread sooner.
Subtracting cost from the result is not the same as charging it to the trigger. The first changes how much you win. The second changes which trades win at all.
Measured: 2–5 percentage points of win rate, always against you.
This generalises: if your simulation applies costs as a post-hoc adjustment rather than modelling the mechanism that generates them, your event counts are wrong, not just your totals.
Bug 2: the win rate was a design choice, not evidence
Targets were 0.5–0.8×ATR, stops 1.5–2.0×ATR. That is a reward:risk of 0.25–0.4, which fixes the breakeven win rate by arithmetic:
breakeven = SL / (TP + SL) = 75–82%
Out-of-sample: 4947 trades, 73.5% win rate, -4669.9 pips, -0.944 pips per trade.
A 73.5% win rate that loses money is not a paradox. Small targets hit often. That is what small targets do.
The diagnostic worth stealing
Before re-optimising anything, I asked whether the parameter surface was learnable at all.
For each walk-forward fold: evaluate the entire parameter grid on the training window and on the test window, then take the Spearman rank correlation between the two.
if rho > 0 -> training rank predicts test rank; selection is meaningful
if rho ~ 0 -> the surface is noise; NO selection rule can help
That second line is the valuable part. It separates "my optimiser is bad" from "there is nothing here" — two situations that look identical from the outside and demand opposite responses.
Result across 119 folds:
- median rho: -0.024
- mean rho: -0.03, 95% CI [-0.09, 0.03] — includes zero
- folds with rho > 0: 47.9% — a coin flip
Zero information. Consistent with that, walk-forward optimisation with plateau smoothing produced worse out-of-sample results than the parameters I never touched, and selected the grid boundary in 11 of 17 configs — the signature of an optimiser with nothing to grip.
Then I checked whether the effect exists anywhere
38 instruments, non-overlapping holding periods, volatility-scaled, pooled by asset class so 1,140 tests collapse to 6, split first-half vs second-half:
| asset class | Sharpe @0bp | breakeven cost | 1st half | 2nd half | stable |
|---|---|---|---|---|---|
| FX major | +0.08 | 3bp | +0.06 | -0.09 | no |
| FX cross | +0.02 | 1bp | -0.05 | -0.06 | no |
| Index | +0.07 | 5bp | +0.04 | +0.02 | yes |
| Metal | +0.12 | 12bp | +0.13 | +0.03 | yes |
| Energy | +0.13 | 23bp | +0.15 | -0.02 | no |
| Crypto | +0.42 | 100bp | +0.54 | +0.15 | yes |
Two things I would not have guessed:
Breakeven transaction cost is more decision-useful than Sharpe. FX majors break even at ~3bp and crosses at ~1bp — both below the actual spread. The question "is there an edge" is settled before you write any strategy code.
The significant result was the fake one. Energy shows t = 3.73 on the full sample. It is flat-to-negative in the second half. Without the split it would have looked like the best finding in the table.
The actual lesson
Build the falsifier before the strategy.
The reason a long series of configurations looked "validated" is that no component in the pipeline had the job of saying nothing here. Every part was designed to find something, so it found something, every time.
The rank-correlation test above took an afternoon and invalidated months of parameter work — including the one lead I was most confident about. That is the test doing its job, and it is the cheapest insurance I have added to a research process in a long time.
