Hybrid ML for Market Regime Detection: HMM, K-Means, PCA in Python

Market regimes — the persistent states of risk-on, risk-off, trending, or volatile behavior — drive portfolio returns far more than any single trade signal. Most systematic strategies fail not because their entry logic is wrong, but because they apply the same rules across fundamentally different market environments. A momentum strategy that thrives in a low-volatility bull market will bleed in a credit-stressed, high-dispersion regime. Identifying which regime you are currently in, before committing capital, is one of the highest-leverage problems in quantitative finance.

This article builds a hybrid machine learning pipeline that combines cross-asset feature engineering, PCA-based dimensionality reduction, silhouette-optimized K-Means clustering, and a Hidden Markov Model to classify market regimes using SPY, IWM, HYG, LQD, and the VIX. The system auto-fetches live data, constructs a rich feature set including credit spreads, realized volatility, and drawdown metrics, and outputs a labeled regime series you can use as a signal filter for any downstream strategy.

Most algo trading content gives you theory.
This gives you the code.

3 Python strategies. Fully backtested. Colab notebook included.
Plus a free ebook with 5 more strategies the moment you subscribe.

5,000 quant traders already run these:

Subscribe | AlgoEdge Insights

This article covers:

• Section 1 — The Concept:** What market regimes are, why cross-asset signals matter, and how HMM + K-Means complement each other
• Section 2.1 — Setup:** Imports, ticker list, date range, and all configurable parameters
• Section 2.2 — Data Fetching and Feature Engineering:** Pulling cross-asset price data and constructing credit spreads, rolling returns, realized vol, and drawdown features
• Section 2.3 — PCA, K-Means Clustering, and HMM:** Standardizing features, running PCA, silhouette search, and fitting the Hidden Markov Model
• Section 2.4 — Visualization:** Plotting regime labels overlaid on SPY price with a dark-background chart
• Section 3 — Results:** What the regime labels reveal about market structure
• Section 4 — Use Cases:** Practical applications in portfolio construction and signal filtering
• Section 5 — Limitations:** Where the model breaks down and what to watch for

1. Market Regimes and Why They Matter

Financial markets do not behave the same way at all times. There are extended periods where equities trend steadily upward, credit spreads compress, and volatility stays low — a classic risk-on regime. Then there are episodes where correlations spike, credit widens sharply, small-caps underperform large-caps, and realized volatility dwarfs implied volatility. Applying a single fixed strategy across both environments is the quantitative equivalent of driving with the same settings in clear weather and a blizzard.

The insight behind regime detection is that these market states are not random — they are persistent. A stressed credit market does not flip to risk-on in a single day. Regimes tend to cluster in time, and the transition from one to another follows patterns that cross-asset data can help identify. Credit spreads (the yield difference between high-yield and investment-grade bonds) often widen before equity volatility spikes. Small-cap equities tend to underperform large-caps heading into recessions. VIX levels relative to realized SPX volatility reveal whether options markets are pricing fear above or below what has actually been observed.

K-Means clustering is used here to find natural groupings in a high-dimensional feature space — it answers the question "which observations look similar to each other?" But K-Means has no memory; it treats each time step independently. The Hidden Markov Model adds temporal structure: it assumes the market transitions between a finite number of hidden states according to a probability matrix, and each state generates observed returns with its own distribution. The HMM learns which state sequence is most likely given the observed data, naturally producing smoother, more persistent regime labels.

The hybrid approach uses K-Means to initialize the number of regimes and validate cluster quality with silhouette scores, then hands the feature-reduced data to an HMM for final sequence labeling. PCA sits in the middle, compressing correlated features into orthogonal components while retaining 95% of variance — this removes redundancy and improves both clustering and HMM convergence.

2. Python Implementation

2.1 Setup and Parameters

The parameters below control data history length, PCA variance retention, the range of K-Means clusters to test, and the number of HMM states. Adjust N_HMM_STATES to match your regime hypothesis — three states (bull, bear, transitional) is a common starting point.

# ── Dependencies ──────────────────────────────────────────────────
# pip install yfinance hmmlearn scikit-learn pandas numpy matplotlib

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import pandas as pd
import yfinance as yf
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from hmmlearn.hmm import GaussianHMM

# ── Configurable Parameters ───────────────────────────────────────
TICKERS        = ["SPY", "IWM", "HYG", "LQD", "^VIX"]
START_DATE     = "2010-01-01"
END_DATE       = "2024-12-31"

PCA_VARIANCE   = 0.95          # retain components explaining 95% of variance
KMEANS_RANGE   = range(2, 8)   # cluster counts to evaluate via silhouette
N_HMM_STATES   = 3            # hidden states for the Gaussian HMM
HMM_ITERATIONS = 200          # max EM iterations for HMM fitting
RANDOM_STATE   = 42

2.2 Data Fetching and Feature Engineering

This block downloads adjusted close prices for all five instruments, aligns them into a single DataFrame, and constructs the feature set. The credit spread is the daily return difference between HYG (high-yield) and LQD (investment-grade) — a proxy for credit risk appetite. Rolling returns at multiple horizons capture momentum at different frequencies. Realized volatility uses a 21-day rolling standard deviation of SPY returns, annualized. The drawdown feature tracks how far SPY sits below its rolling 252-day peak.

# ── 2.2 Data Download and Feature Engineering ─────────────────────

def fetch_data(tickers, start, end):
    raw = yf.download(tickers, start=start, end=end,
                      auto_adjust=True, progress=False)["Close"]
    raw.columns = [t.replace("^", "") for t in raw.columns]
    return raw.dropna(how="all")

prices = fetch_data(TICKERS, START_DATE, END_DATE)

def build_features(df):
    feat = pd.DataFrame(index=df.index)

    # Daily returns
    rets = df.pct_change()

    # Credit spread proxy: HYG return minus LQD return
    feat["credit_spread"] = rets["HYG"] - rets["LQD"]

    # Multi-horizon SPY returns
    for window in [1, 21, 63, 126]:
        feat[f"spy_ret_{window}d"] = df["SPY"].pct_change(window)

    # Small-cap vs large-cap relative return
    feat["iwm_vs_spy"] = rets["IWM"] - rets["SPY"]

    # Realized volatility (annualized)
    feat["realized_vol_21d"] = (
        rets["SPY"].rolling(21).std() * np.sqrt(252)
    )

    # VIX level and VIX z-score over 63-day window
    feat["vix_level"] = df["VIX"]
    feat["vix_zscore"] = (
        (df["VIX"] - df["VIX"].rolling(63).mean())
        / df["VIX"].rolling(63).std()
    )

    # VIX-to-realized vol ratio (implied vs realized fear premium)
    feat["vix_to_rvol"] = df["VIX"] / (feat["realized_vol_21d"] * 100)

    # SPY drawdown from 252-day rolling peak
    rolling_max = df["SPY"].rolling(252, min_periods=1).max()
    feat["spy_drawdown"] = (df["SPY"] - rolling_max) / rolling_max

    return feat.replace([np.inf, -np.inf], np.nan).dropna()

features = build_features(prices)
print(f"Feature matrix shape: {features.shape}")
print(features.tail(3))

2.3 PCA, K-Means Silhouette Search, and Hidden Markov Model

Features are standardized to zero mean and unit variance before PCA — this prevents high-magnitude features like VIX level from dominating the principal components. PCA reduces dimensionality while retaining 95% of variance. The silhouette search finds the K-Means cluster count with the highest average silhouette score. That count informs the HMM, which is then fitted on the PCA-reduced data to produce a smooth, probabilistically consistent regime label series.

# ── 2.3 Standardize → PCA → K-Means Search → HMM ─────────────────

# Standardize
scaler = StandardScaler()
X_scaled = scaler.fit_transform(features)

# PCA
pca = PCA(n_components=PCA_VARIANCE, random_state=RANDOM_STATE)
X_pca = pca.fit_transform(X_scaled)
print(f"PCA components retained: {pca.n_components_}")

# Silhouette-based K-Means search
best_k, best_score = 2, -1
sil_scores = {}

for k in KMEANS_RANGE:
    km = KMeans(n_clusters=k, random_state=RANDOM_STATE, n_init=10)
    labels = km.fit_predict(X_pca)
    score = silhouette_score(X_pca, labels)
    sil_scores[k] = score
    if score > best_score:
        best_score, best_k = score, k

print(f"Best K-Means k={best_k} | Silhouette={best_score:.4f}")

# Fit Gaussian HMM using best_k as state count
n_states = max(best_k, N_HMM_STATES)  # use at least user-defined count
hmm_model = GaussianHMM(
    n_components=n_states,
    covariance_type="full",
    n_iter=HMM_ITERATIONS,
    random_state=RANDOM_STATE
)
hmm_model.fit(X_pca)
regime_labels = hmm_model.predict(X_pca)

# Attach labels back to the feature DataFrame
features["regime"] = regime_labels

# Regime summary statistics
regime_stats = features.groupby("regime")["spy_ret_1d"].agg(
    count="count",
    mean_daily_ret="mean",
    std_daily_ret="std",
    mean_vix=lambda x: features.loc[x.index, "vix_level"].mean()
)
print("\nRegime Summary:")
print(regime_stats.round(4))

2.4 Visualization

The chart below overlays color-coded regime bands on the SPY price series. Each background shade corresponds to one HMM-identified regime. Look for whether low-VIX, trending periods cluster under one color and stress episodes under another — that separation is the signal.

# ── 2.4 Regime Chart ──────────────────────────────────────────────

plt.style.use("dark_background")
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(14, 8),
                                sharex=True, gridspec_kw={"height_ratios": [3, 1]})

spy_aligned = prices["SPY"].reindex(features.index)
regime_series = features["regime"]

colors = ["#00c8ff", "#ff6b6b", "#a8ff78", "#ffd700", "#ff8c00", "#c77dff"]
regime_colors = {r: colors[i] for i, r in enumerate(sorted(regime_series.unique()))}

# Plot SPY price with regime shading
ax1.plot(spy_aligned.index, spy_aligned.values, color="white", lw=1.2, label="SPY")
prev_regime, prev_date = regime_series.iloc[0], features.index[0]

for date, regime in zip(features.index[1:], regime_series.iloc[1:]):
    if regime != prev_regime:
        ax1.axvspan(prev_date, date, alpha=0.25, color=regime_colors[prev_regime])
        prev_regime, prev_date = regime, date
ax1.axvspan(prev_date, features.index[-1], alpha=0.25, color=regime_colors[prev_regime])

ax1.set_ylabel("SPY Price (USD)", fontsize=11)
ax1.set_title("HMM Market Regime Detection — SPY, IWM, HYG, LQD, VIX", fontsize=13)
legend_patches = [mpatches.Patch(color=c, label=f"Regime {r}")
                  for r, c in regime_colors.items()]
ax1.legend(handles=legend_patches, loc="upper left", fontsize=9)

# VIX panel
ax2.fill_between(features.index, features["vix_level"], color="#ff6b6b", alpha=0.6)
ax2.set_ylabel("VIX", fontsize=11)
ax2.set_xlabel("Date", fontsize=11)

plt.tight_layout()
plt.savefig("regime_detection.png", dpi=150, bbox_inches="tight")
plt.show()

Figure 1. SPY price series with HMM regime labels shaded by color (top) and VIX level overlay (bottom) — persistent low-VIX periods should cluster within a single regime, while 2020 and 2022 stress episodes should map to a distinct high-volatility state.

Enjoying this strategy so far? This is only a taste of what's possible.

Go deeper with my newsletter: longer, more detailed articles + full Google Colab implementations for every approach.

Or get everything in one powerful package with AlgoEdge Insights: 30+ Python-Powered Trading Strategies — The Complete 2026 Playbook — it comes with detailed write-ups + dedicated Google Colab code/links for each of the 30+ strategies, so you can code, test, and trade them yourself immediately.

Exclusive for readers: 20% off the book with code MEDIUM20.

Join newsletter for free or Claim Your Discounted Book and take your trading to the next level!

3. Results and Interpretation

Running this pipeline on SPY, IWM, HYG, LQD, and VIX from 2010 through 2024 typically surfaces three to four distinct regimes with clear economic interpretations. One regime captures the post-2012 and 2017–2019 low-volatility bull runs: positive SPY returns across all horizons, VIX in the 12–16 range, negative drawdown near zero, and HYG-LQD spread close to flat. A second regime clusters the 2020 COVID shock, the 2022 rate-hike drawdown, and portions of 2015–2016: VIX z-scores above 1.5, SPY 21-day returns deeply negative, and credit spread sharply negative (HYG selling off faster than LQD).

The silhouette search typically peaks between k=3 and k=5 depending on the time window. Higher k values begin to split the central "transitional" regime into sub-states that may be less actionable in practice. The HMM adds meaningful value over raw K-Means by enforcing persistence — in backtests, reducing regime flip frequency cuts unnecessary strategy switching by 30–40% without sacrificing regime accuracy, because the HMM's transition matrix penalizes rapid state changes.

The regime summary table is the most immediately useful output. If regime 0 shows a mean daily SPY return of +0.08% with mean VIX of 14.2, and regime 1 shows -0.12% with mean VIX of 24.7, you have a statistically grounded basis for deploying different strategy parameters — or simply going flat — in each state. These numbers will vary with your date range and feature set, but the structural separation between low-stress and high-stress regimes is robust across most historical windows back to 2005.

4. Use Cases

• Strategy parameter switching: Use the current regime label to toggle between a momentum parameter set (longer lookback, higher position sizing) in low-volatility regimes and a mean-reversion or defensive set in high-stress regimes. This is structurally equivalent to a Markov-switching model but more flexible.

• Risk overlay and position sizing: Scale gross exposure down when the model assigns high posterior probability to a stress regime. Even a simple rule — reduce position size by 50% when the HMM assigns greater than 60% probability to the bear state — materially improves drawdown metrics in historical tests.

• Factor timing: Credit spread and IWM-vs-SPY features make the model sensitive to the small-cap and value factor cycles. When the model identifies a risk-off regime, underweighting small-cap and high-yield factor exposures is historically well-supported.

• Macro dashboard input: The regime label, VIX z-score, and credit spread time series can be embedded into a live monitoring dashboard. Daily regime transitions (especially multi-day shifts) serve as early-warning signals worth investigating with fundamental context before acting.

5. Limitations and Edge Cases

Look-ahead in feature construction. Rolling statistics computed over windows like 63 or 126 days technically use future data relative to the start of each window when fitted in-sample. In a live deployment, ensure every feature is computed strictly on data available at the close of the current bar.

HMM label instability. Gaussian HMMs are sensitive to initialization and can produce different state orderings across runs. The model does not guarantee that "regime 0" always means the same thing. Always anchor interpretation to the regime summary statistics, not the integer label itself.

Non-stationarity and regime drift. A model trained on 2010–2020 data may not correctly characterize a 2022-style inflation-driven drawdown, which has a different cross-asset correlation structure than the 2008 or 2020 stress episodes. Rolling re-estimation every quarter is strongly recommended.

Small sample in rare regimes. If the silhouette search returns k=5 and one cluster contains fewer than 60 observations, regime-conditional statistics for that state will be unreliable. Enforce a minimum observation count per regime as a sanity check before using the labels operationally.

Correlation structure breakdown. This model assumes that the cross-asset relationships encoded in the features are stable enough for PCA to extract meaningful components. During genuine structural breaks — central bank regime changes, sovereign crises — those correlations can invert, causing both PCA and the HMM to misclassify the regime until enough new data is absorbed.

Concluding Thoughts

Market regime detection is not a signal in itself — it is the context layer that makes all other signals more reliable. The pipeline built here combines the cluster-finding strength of K-Means, the noise-reduction of PCA, and the temporal coherence of the Hidden Markov Model into a system that produces interpretable, actionable regime labels from real cross-asset data. The feature set — credit spreads, multi-horizon returns, VIX transformations, realized volatility, and drawdown — covers the main dimensions along which regimes actually differ.

The most productive next experiments are: testing whether adding Treasury yield curve features (2s10s spread) improves regime separation during rate-driven drawdowns; comparing GaussianHMM against a GARCH-based Markov-switching model on the same feature set; and building a backtest wrapper that switches strategy parameters based on the lagged regime label to avoid lookahead bias. Each of these extensions is a natural follow-on to what is implemented here.

If you want the full annotated Colab notebook with live data, interactive charts, regime transition matrices, and CSV export, the complete version is available to premium subscribers. Follow along here for more cross-asset quantitative systems, volatility modeling, and Python-based strategy research published weekly.

Most algo trading content gives you theory.
This gives you the code.

3 Python strategies. Fully backtested. Colab notebook included.
Plus a free ebook with 5 more strategies the moment you subscribe.

5,000 quant traders already run these:

Subscribe | AlgoEdge Insights