I gave Claude Code ten years of USD/JPY daily bars and asked it to find a trading rule. It ran backtests through an API, as many as it wanted up to 30, but the last 20% of the data was hidden from it the whole time. At the end it got exactly one look at the hidden part, and it had to write down what it expected before it looked.
The point wasn't to find a strategy. It was to see what happens to "the best backtest an AI found" when the AI can't quietly try its way into it.
The setup
- Data: USD/JPY daily, April 2015 to April 2025, 2,604 bars. The last 520 bars (from 2023-04-26) were cut off on the server, so the agent's runs never received those prices.
- Agent: Claude Opus 5.5 in Claude Code, 2026-10-19, working through Hawk Backtester's Agent API. It wrote each strategy as a Python file and ran it with a small script.
- Rules: at most 30 search runs. After each run, a note saying what it changed and why. When done, pick one strategy, write a prediction for the hidden period, then run it once on the hidden period.
- The ledger: every run goes into one study. Each result comes back with the trial number and a deflated Sharpe ratio: the probability that the run beats the best you'd expect from that many tries of pure noise.
What it tried
It used 13 of its 30 runs and then stopped on its own: "further searching would mostly have added selection bias."
| # | Rule | Visible return | Max DD | Trades | Deflated Sharpe |
|---|---|---|---|---|---|
| 1 | 50-day breakout, fixed stop and target, 1,000 units | −53.9% | −55% | 3 | 0.13 |
| 2 | same, sized to risk 2% per trade | +14.6% | −16.2% | 35 | 0.06 |
| 3 | 50-day breakout, 3-yen trailing stop, no target | +12.0% | −7.5% | 42 | 0.003 |
| 4 | trailing stop scaled to recent volatility | −7.3% | −21.4% | 75 | 0.0 |
| 5 | #3 plus a 200-day trend filter | +6.2% | −9.5% | 37 | 0.002 |
| 6 | 6-month momentum, always in the market | +14.6% | −21.0% | 62 | 0.0 |
| 7 | hold when 3 of 4 lookbacks agree | −2.4% | −26.2% | 123 | 0.0 |
| 8 | 12-month momentum, checked monthly | +14.4% | −15.7% | 16 | 0.04 |
| 9 | buy 5-day lows above the 200-day average | −5.5% | −17.1% | 119 | 0.0 |
| 10–13 | #3 with a 100-day / 20-day lookback, 2-yen / 4-yen trail | −3.3% to +9.9% | 32–73 | ≤ 0.007 |
Run 1 was a sizing bug, not a test of the signal, and the agent said so in its next note. Runs 10 to 13 weren't new ideas; it was checking whether #3 sat on a plateau or a spike. The lookback held up (20, 50 and 100 days were all positive). The trailing-stop distance didn't (2 yen lost money, 4 yen made a third as much), and it noted that "3 yen is partly a lucky value."
Here is the dashboard's research note at the end. The labels are Japanese: hidden period 20% (520 bars), hidden period used once, 13 trials, and the run list with each trial's deflated Sharpe in the last column.
What the ledger said
Every search run came back "within chance, given the number of trials." The luck threshold (the per-trade Sharpe you'd expect from the best of N noise strategies) jumped to about 0.5 by the third trial and stayed between 0.47 and 0.53. The pick, #3, had a per-trade Sharpe of 0.127 and a deflated Sharpe of 0.003.
The agent agreed with it. From its notes before the final run:
Ledger says chance, and I agree: I expect roughly break-even, not a confirmation.
The prediction, then the hidden period
Written before the run: about +1% over the hidden two years (range −6% to +8%), max drawdown about 6%, about 10 trades, shorts losing money again.
What happened:
- Equity over the hidden period: +1.2% (11,201 at the cut to 11,334), max drawdown −7.4%.
- Trades entered in the hidden period: 24, net +245, profit factor 1.31, win rate 46%, per-trade Sharpe 0.065 (about half the visible 0.127).
- Trade count: missed. It expected about 10 and got 24. The hidden period had far more breakouts and stop-outs than the visible average.
Its own summary:
The pick didn't fall apart, but it didn't prove anything either. It earned a small positive return in the hidden period, per trade about half as good as on the visible data. That fits a weak edge, or no edge plus some luck. I would not call this a working strategy.
What I take from it
The interesting part isn't the strategy. It's that the agent didn't oversell it. Without the ledger, the story writes itself: "AI finds a USD/JPY breakout system, +12% with a 7.5% drawdown." With the ledger in every response, the agent kept reading "within chance," stopped searching early, ran sensitivity checks instead of new ideas, and predicted the hidden period close to what happened.
Two things I changed after watching it:
-
The holdout response mixed the periods. A holdout run reported a total return of +13.3% because the outcome covered the whole run. The agent noticed and worked out the hidden part itself. Holdout runs now also return
research.hidden_periodwith the equity change and drawdown for the hidden period alone. -
Agent Mode was broken in production before this run (the browser runner rejected every session with "runId is missing"), and a failure in the browser came back to the agent as an empty success. Both are fixed: a browser-side failure now comes back as
failedwith the message, and a run no browser picks up says so after 30 seconds.
Limits
- One dataset, one period, one model. Daily bars, no swap or carry costs.
- The model's training data includes general knowledge of USD/JPY history up to 2025. It said so in its summary, and its picks followed the visible runs, but I can't rule out that it helped.
- A 2-year holdout can't confirm a weak edge. All this shows is that this pick wasn't an overfit illusion that collapsed.
The ledger, the hidden period and the single holdout are part of Hawk Backtester's Agent Mode. Strategies are plain Python files that run on your machine (pip install hawk-bt).
If you let agents run backtests, what do you use to keep them from picking the lucky run?
