[Oct 17] Ending a 60-Day Shadow Experiment, By the Rule I Set in Advance

An experiment that didn't clear its pre-registered statistical bar got retired on schedule, and a data correction plus an old incident's follow-up got closed out the same day.

This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).

Closing a 60-day experiment by the rule I wrote down in advance

For a while now I'd been running a position-sizing variant in shadow mode only — never on the live account — for more than 60 days straight.

When I started this experiment, I wrote down in advance what counted as "clearly better, adopt it," "ambiguous, keep watching," and "below this line, drop it." The point was to fix the bar before seeing the numbers, so I couldn't bend the rule to fit whatever came out.

Today I had another AI recompute the full 60 days of data and check it against that pre-registered bar. The net-return edge over the existing approach wasn't statistically convincing enough, and it also fell short of the threshold for an extension.

I also ran the numbers on what it would take to rescue the extension case, and at the current size of the gap it would take years more data to get there. At that point it's not "ambiguous, keep watching" anymore — it's just "didn't clear the bar." So I retired the experiment.

This variant had only ever lived on the shadow track, so the live account was never affected. Retiring it only changes its documented status — the shadow observation itself costs nothing, so I left it running to keep watching anyway.

Closing out a price-data correction and an old driver incident's follow-up

I finished a price-data correction job I'd been chipping away at for several days. It backfills tiny discrepancies that show up once the exchange's finalized prices arrive later than the provisional ones — widening the backfill window got every stock in production to match the finalized figures.

I also separately checked whether this discrepancy had actually affected real trades.

The conclusion: there was some effect, but at a level too small to detect reliably. So I didn't rescore anything or recompute past performance — just logged it for the record.

And I closed out the follow-up to an incident from a few days back, where an unattended system update had touched the GPU driver and nearly stalled an overnight job. The relevant packages are now excluded from automatic updates.

Also today

I went through the whole TODO document and pulled out follow-up items that had been buried inside already-closed lines. None of them are urgent, but I logged them so they don't get lost.

Another signal came back with an ambiguous statistical result, so I decided to leave it as-is for now and check again in about a month.

Why this matters

What mattered most today wasn't the decision to retire the experiment itself — it was that the call followed a rule set in advance.

When data comes back ambiguous, the natural pull is to keep watching a little longer. Writing the bar down beforehand cuts against that pull.

The data correction and the driver-incident follow-up carry the same logic. The job isn't done when you patch the immediate problem — it's done once you've traced the full blast radius and closed the loop on preventing it from happening again.