Turning the CFTC Commitments of Traders report into a weekly crowd-positioning gauge

Every Friday afternoon, the US Commodity Futures Trading Commission (CFTC) publishes who holds what in the US futures markets. It's called the Commitments of Traders report, or COT, and it has been around for decades. Most people read it for what the big players are doing. I wanted the other end: how the small accounts are positioned, and whether that's unusual for them.

This post covers how we turned that into a weekly 0 to 100 reading we call the Crowd Gauge: the source, the fields, the percentile method, a data quirk that bit us, and the limits.

The source: a public API, no key

The CFTC serves COT data through a Socrata API at publicreporting.cftc.gov. No sign-up, no key. (An optional app token raises the rate limit; we read one from the environment if it's set.)

We use the Legacy, futures-only report, dataset id 6dca-aqww. It covers stock indices, metals, energy, currencies and the CME crypto contracts. Each row is one contract market on one report date.

The request is a plain filtered query. This is a cut-down version of our collector, with the same parameters:

import json
import urllib.parse
import urllib.request

BASE = "https://publicreporting.cftc.gov/resource/6dca-aqww.json"
DATE_COL = "report_date_as_yyyy_mm_dd"
CODE_COL = "cftc_contract_market_code"

codes = ["13874A", "088691", "133741"]  # E-mini S&P 500, gold, CME bitcoin
where = f"{CODE_COL} in({','.join(repr(c) for c in codes)})"
where += f" AND {DATE_COL} >= '2023-01-01T00:00:00'"

params = {"$where": where, "$order": f"{DATE_COL},{CODE_COL}",
          "$limit": 50_000, "$offset": 0}
url = BASE + "?" + urllib.parse.urlencode(params)

with urllib.request.urlopen(url, timeout=120) as r:
    rows = json.load(r)

The real collector pages 50,000 rows at a time, keeps a bookmark so later runs fetch only new weeks, and stores each raw response with a SHA-256 hash. That hash is in our published JSON, so any reading traces back to the exact bytes it came from.

The fields we actually use

COT splits open interest into reportable traders (big enough to hit the CFTC's reporting thresholds) and non-reportable positions: everything left over. Non-reportable is the closest thing the report has to "the retail crowd". So from each row we keep just four things:

From those we compute one raw number per contract per week, the long share:

share = 100.0 * long / (long + short)

On its own that number is hard to read. On the 6 October 2026 report, small traders in the E-mini S&P 500 held 248,117 long against 137,112 short, a 64.41% long share. Is that a lot? Every market has its own built-in tilt, so 64% can be normal in one contract and extreme in another.

Release date, not position date

One detail that's easy to get wrong in a backtest: the report describes Tuesday's positions but nobody can see it until Friday. We date every value from its release, three days after the position date:

COT_RELEASE_LAG = 3  # Tuesday as-of -> Friday release
released = as_of + timedelta(days=COT_RELEASE_LAG)

Holiday weeks sometimes slip a day or two later than that. We accept that small error rather than maintain a holiday calendar [judged].

The 3-year percentile

To strip out each market's normal lean, we rank this week's long share against the same contract's previous 156 weekly readings (three years). Ties count half. A contract needs at least 52 readings before it produces a value at all.

def pct_rank(window: list[float], x: float) -> float:
    below = sum(1 for v in window if v < x)
    equal = sum(1 for v in window if v == x)
    return 100.0 * (below + 0.5 * equal) / len(window)

WINDOW, MIN_OBS = 156, 52
for i, o in enumerate(obs):
    win = shares[max(0, i - WINDOW + 1): i + 1]
    o["pct"] = pct_rank(win, shares[i]) if len(win) >= MIN_OBS else None

This is the long-standing "COT index" idea; we didn't invent it. Percentiles also cope with fat tails better than z-scores, which matters when one week can be several times bigger than the rest.

The E-mini S&P 500's 64.41% long share ranked at 76.6. That means it was higher than about three-quarters of the last three years.

From there it's averaging:

1. Market = mean of its contracts (bitcoin and ether each have a standard and a micro contract).
2. Class = equal-weight mean of its markets. Futures: E-mini S&P 500, Nasdaq-100, gold, WTI crude, euro, yen. Crypto: bitcoin, ether.
3. Composite = 50% futures, 50% crypto. These weights were fixed before any testing and are never tuned.

A COT value stays usable for up to 10 days, then drops out. If less than half the composite weight is live, no composite is published that day.

Readings fall into five bands: 0–20 crowded short, 20–40 leaning short, 40–60 balanced, 60–80 leaning long, 80–100 crowded long. On the 9 October 2026 release the composite read 65.62 (leaning long): futures 55.77, crypto 75.48. You can see the current values on the live Crowd Gauge.

Pitfall: the CME quarterly roll

This one we caught late. In the published observations, the report dated 15 September 2026 looks odd for several CME currency futures. Mexican peso non-reportable longs jump to 68,934, against a median of 7,637 over the previous 12 weeks. New Zealand dollar longs jump to 48,402 against 3,206. Shorts balloon the same way, so the long share falls towards 50%: the peso went from 69.03% the week before, to 51.46%, and back to 71.21% the week after. Its percentile dropped from 51.6 to 24.68 for one week.

The same spike shows up on 16 December 2025 and 16 June 2026. Those dates fall in the quarterly expiry weeks of CME currency futures, when positions in the old and new contract months are open at the same time. Our reading is that this is a roll effect, not a real shift in sentiment [judged].

Method v1.0 doesn't filter these weeks. Old readings are never rewritten, so a fix will come as a new method version. A quick way to flag them yourself:

import pandas as pd

obs = pd.read_csv("crowd-gauge-observations.csv")
obs["median_12w"] = (obs.groupby("leg")["nonrep_long"]
                        .transform(lambda s: s.shift(1).rolling(12).median()))
obs["spike"] = obs["nonrep_long"] / obs["median_12w"]
print(obs[obs["spike"] > 3][["market", "as_of_date", "nonrep_long", "median_12w"]])

On the file as published (data to 9 October 2026), a 3× threshold flags 19 rows. Sixteen of them fall on those three dates, all in CME currency contracts. March 2026 shows no spike, and almost nothing like it appears before late 2025. We haven't explained either yet, so treat the roll explanation as a working theory.

We wrote the test down first

We wanted to know whether extreme readings (80 and over, or 20 and under) came before unusual returns. So we committed the test rules to the repository before running it: horizons, periods, statistics, and the exact pass/fail wording. We call it "an edge" only if the pooled 4-week coefficient from 2018 onwards has |t| ≥ 2 with Newey-West errors, and points the same way in 2006–2017.

The result (measured, across 14 markets with free price series):

• Crowded long: passed. Four-week returns after 80+ readings were 1.0 percentage point above each market's average from 2018 onwards (t = 2.54). In 2006–2017 it was +0.30 (t = 1.18). That's the opposite of the popular "fade the crowd" idea.
• Crowded short: no reliable edge (t = 1.49).

The caveats matter as much as the result. The effect is small next to normal price swings. Leaving out bitcoin and ether (a check we added afterwards, so not part of the rule) drops the crowded-long t-statistic to 1.97. Pooling also treats markets as independent, which flatters the t-statistics. A pattern in past data may not repeat.

Honest limitations

• "Retail" is a proxy. Non-reportable means "below the CFTC's reporting size", which includes some small firms.
• It's weekly and three days old. Until our exchange data sources qualify (they need 180 days of collection first), the gauge moves once a week.
• Futures only. Retail spot forex and CFD positioning sits with brokers, and we only use that with their permission.
• A percentile is relative. A reading of 90 means "more long than in 90% of the last three years", not "90% of traders are long".
• Roll weeks, as above.

Get the data

Everything is CC BY 4.0. The weekly observations CSV has the raw non-reportable longs and shorts, long share and both percentiles for every contract since 2006. The full method write-up has the backtest tables by market.

This is data about positioning, not investment advice. Futures and crypto derivatives are leveraged and carry a high risk of losing money.

Sydney Stones is a co-founder of TradingLookup and leads its data and method work.