I Let an LLM Agent Search for Trading Strategies. A Simple Heuristic Beat It.

I gave DeepSeek V3 forty chances to find a profitable trading strategy.

A simple heuristic found +52% net. The LLM's early proposals lost money. After fixing three silent bugs the agent surfaced, it improved — then optimized its way to +123.7%, a number that didn't survive scrutiny. Here's what happened across four rounds.

The setup

In earlier posts in this series I tested six dead-simple strategies and three more sophisticated edges — all killed by fees, adverse selection, or data leakage. What survived was a backtest harness: point-in-time universe, walk-forward, trading costs, out-of-sample scoring only. Same rules for every strategy, no exceptions. I put an agent in front of it.

A proposal is a typed JSON object:

@dataclass
class Proposal:
    family: str        # "reversal" | "factor_residual" | "momentum"
    bar_size: str      # "15m" | "1h" | "1d"
    lookback: int      # days
    holding_period: int
    entry_threshold: float

# The agent loop
def agent_loop(n_rounds: int = 40) -> list[ScoredProposal]:
    history: list[ScoredProposal] = []
    for _ in range(n_rounds):
        prompt = build_prompt(history)          # what failed, what survived
        json_str = llm(prompt)                  # DeepSeek V3 via OpenRouter
        proposal = parse_proposal(json_str)
        verdict = harness.score(proposal)       # walk-forward over 451 symbols, 76 windows
        history.append(ScoredProposal(proposal, verdict))
    return history

The agent never touches the scoring. It proposes; the same judge that every hand-built strategy went through decides. There is no way for it to talk its way to a good number.

I also kept a heuristic proposer alongside it: samples the space randomly, occasionally perturbs the current best. It is not purely random — the perturbation step is local optimization — but it needs no model and costs nothing. Both the heuristic and the agent ran for the same 40 evaluation rounds. That baseline turned out to matter a lot.

Round one: the model lost to the heuristic

The first LLM run was worse than the heuristic search.

DeepSeek had the right instinct — it leaned toward factor-residual reversion on daily bars (buy coins whose recent return lagged the cross-sectional average, hold for ~30 days), which was the only corner that remained promising in this run. But it picked short lookbacks and fast rebalances:

factor_residual/1d/lb30/reb15   -23.7%   reject
factor_residual/1d/lb20/reb10   -44.4%   reject
reversal/15m/lb5/reb2           -99.1%   reject

Meanwhile the heuristic had stumbled into factor_residual/1d/lb60/reb30: about +52% net, positive in most out-of-sample windows, tiny turnover. The model landed one step away from the edge and picked the wrong side of it. Right family, wrong calibration. The heuristic beat it.

The bugs the harness caught

Before I wrote the model off, something more useful turned up.

Several proposals came back with zero turnover and zero return. Not negative — flat. The strategy had never opened a position. The three bugs had the same symptom: a flat book, zero turnover, no crash, and no error message. Just silence.

Bug 1: Momentum's volatility window was scaled wrong.
The signal normalized by a 96-bar trailing volatility. On 15-minute data that is one day — fine. On daily bars it is 96 days, longer than the formation window, so the signal never became valid.

# Bug: hardcoded window ignores bar size
vol_window = 96

# Fix: scale to timeframe; floor of 5 bars avoids a 1-bar window on daily
BAR_MINUTES = {"15m": 15, "1h": 60, "1d": 1440}
vol_window = max(5, round(96 * 15 / BAR_MINUTES[bar_size]))
# 15m → 96 bars (one day)  |  1h → 24 bars  |  1d → max(5,1) = 5 days

Bug 2: Lookback could exceed the formation window.
The agent proposed a 100-day lookback on a 90-day formation window. Fix: clamp the lookback to what is actually available.

lookback = min(proposal.lookback, formation_window - 1)

Bug 3: The rebalance grid starved itself.
The wrapper refreshed positions on a fixed grid from bar zero: bar 0, bar 40, bar 80. If the signal only came alive around bar 90, every refresh point landed in the warmup period. A holding period of 40 was dead while 30 and 45 traded — non-monotonic results that mean "bug", not "no edge".

# Bug: grid anchored at bar 0
rebalance_bars = range(0, n_bars, holding_period)

# Fix: anchor at first live bar
first_live = warmup_length
rebalance_bars = range(first_live, n_bars, holding_period)

None of these were crashes. Each returned a flat book with no error. That is exactly why a harness should fail loudly when a strategy that is meant to trade has zero turnover. Treating "no position" as a valid zero-return result would have buried all three.

Round two: it maps the edge

With the space cleaned up, I reran the LLM agent. This time it clustered on factor-residual daily and swept the lookback itself:

factor_residual/1d/lb70/reb35   +67.1%
factor_residual/1d/lb50/reb25   +65.9%
factor_residual/1d/lb40/reb20    +6.5%
factor_residual/1d/lb30/reb15   -26.6%

The model drew the line itself. Long lookback positive, short lookback negative, and it found the boundary from feedback alone — no knowledge of what the heuristic had found. Right family, right direction, just needed a clean space to work in.

So is the edge real? Not established.

Across lookbacks from 40 to 80 days with monthly rebalance it stays net positive, roughly +46% to +84% over the backtest period. (The edge map above shows lb40 at only +6.5% because that sweep used a faster reb20 rebalance — different configuration, same lookback.) Positive in four of five years, down hard in 2023. Survives a pessimistic 30 bps per side. Annualized Sharpe around 0.9. A label-permutation null — which scrambles which coin gets which cross-sectional residual, testing whether the ranking actually carries signal — puts it at p = 0.048, right at the edge of significant.

A promising but unconfirmed lead. The parameters and family were selected on these same results, so the p-value doesn't account for the full search process. Not a confirmed strategy — the backtest window is used up.

Round three: a better optimizer, a bigger number

I wrapped the model in a hybrid proposer — the LLM picks the family and timeframe, but on alternating turns it sweeps the lookback and rebalance around the current best.

It found a peak at a 50-day lookback with monthly rebalance: +123.7% net.

(Round 2 showed lb50 at +65.9% with reb25. Same lookback, different rebalance, and now optimizing both jointly — that's where the extra 60 points came from. Also the best of 35 configs, not the average.)

This is a trap, and one column gives it away:

* 20 bootstrap runs is preliminary; treat the positive-run counts as directional, not precise.

Magnitude doubled. Breadth barely moved.

You can inflate magnitude by picking better parameters. Breadth doesn't work that way. It didn't move.

Here's the problem with reporting the best of 35 configs: even if none of them had any real edge, random variation across 35 tries would still produce one winner. A deflated Sharpe helps, but only for the search you tell it about. The reported 0.70 used this local 7×5 Round 3 grid: N = 35. It did not include the earlier LLM rounds, the heuristic search, the family/timeframe choice, or the debugging runs that led to the grid. Nor are these 35 independent tests: nearby lookbacks produce highly correlated variants of the same signal.

So 0.70 is not a fully search-corrected significance claim. It is a local-grid diagnostic — and even under that friendlier accounting it missed the project's 0.95 qualification threshold. The honest conclusion is simpler: the best configuration did not establish significance. A broader accounting of the search would make the evidence weaker, not stronger.

Round four: I hadn't used my own tools

Overfitting has known cures. I hadn't reached for any of them.

Stop picking a winner. Average the whole plateau instead.

I took all factor-residual configs on daily bars with lookback between 40 and 80 — the range where the sign stayed consistent — and averaged them without picking a winner.

The plateau range itself came from inspecting these same results, so it doesn't fully eliminate selection bias. But it does reduce sensitivity to any single lucky configuration. The +75.8% is less sensitive to one lucky setting, but it is still an in-sample research result, not independent confirmation of alpha.

# Plateau: family=factor_residual, bar=1d, lookback in [40..80]
# Range chosen where sign was consistent — but still data-derived, not pre-specified.
plateau = [r for r in results
           if r.family == "factor_residual"
           and r.bar_size == "1d"
           and 40 <= r.lookback <= 80]

ensemble_signal = np.mean([r.signal for r in plateau], axis=0)

I equal-weighted the factor-residual signal across lookbacks from 40 to 80 days, one combined strategy, monthly rebalance.

Result: +75.8% net, annualized Sharpe 0.61, positive in 62% of windows. That is not a significance claim. A rough calculation using five annual observations puts the t-statistic at about 1.4, before any search correction. The 30-day holds also make adjacent walk-forward windows overlap. So 62% positive windows is a useful breadth check, not 76 independent confirmations.

Both configurations target the same hypothesized signal. The ensemble is less exposed to one lucky (lb, reb) combination; that is all the bootstrap difference shows.

The point

I did not build an AI that finds alpha.

I built a cheap agent with the right instinct and the wrong calibration. The harness exposed three silent bugs because a flat book looked flat. The optimizer then used the cleaned-up space to add 60 percentage points to a backtest number. The ensemble made the remaining signal less tied to one parameter choice, but did not turn it into proof.

Three things I'd take from this:

Run a heuristic baseline first. Until the search space was clean, it beat the model. Don't skip it just because the LLM reasons better.

The agent is only as honest as the thing that scores it. The judge comes first. The proposer is easy to swap out.

Once the proposer can optimize, it will. Watch breadth, not magnitude. And when you can average the plateau instead of picking the best config, do it — the gap between +124% and +76% was entirely parameter sensitivity.

Give a language model a truthful referee and a constrained enough space, and it can find a corner worth inspecting. It will not necessarily beat a simple heuristic, especially if the search space has silent bugs. Once it can optimize, expect a number larger than the evidence warrants unless you correct for the search.

The referee is the part that matters.