Why Your Nifty XGBoost Model Overfits (and How to Fix It with Walk-Forward Validation)

You trained an XGBoost model on Nifty 50 option-chain data. Training AUC = 0.94. Real-money results? Negative P&L within two weeks.

Surprise: your model didn’t fail. It overfitted — and the fix is simpler than switching to a deeper neural net.

The Overfitting Trap in Retail ML

Overfitting happens when a model memorizes noise instead of learning signal. In finance, noise is abundant:

• Expiry-day theta spikes
• Date-wise holiday gaps
• Broker-specific data delays
• Random RBI/Political tweets moving markets

XGBoost is powerful, but it’s also eager to please. Give it enough features and enough trees, and it will fit the training data perfectly — including the noise.

Symptoms of overfitting in Nifty models:

1. Perfect backtest, flat live P&L
2. Feature importance changes every re-train
3. Model works on 2023 data, fails on 2024
4. Win rate collapses during volatile sessions

Walk-Forward Validation: The Standard Most Retail Coders Skip

Walk-forward validation (WFV) respects the temporal structure of markets. Instead of random train/test splits, you:

• Train on Jan-Mar, test on Apr
• Train on Jan-Apr, test on May
• Train on Jan-May, test on Jun
• And so on…

This mimics real life: you never have future data when you trade today.

Python snippet (pandas + sklearn):

import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

# Assume df has columns: date, strike, iv, oi, volume, target (next-day move)
df = df.sort_values('date')
X = df[['strike','iv','oi','volume']]
y = df['target']

tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
    X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
    y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
    model = xgb.XGBClassifier(n_estimators=200, max_depth=4, learning_rate=0.05)
    model.fit(X_train, y_train)
    score = model.score(X_test, y_test)
    print(f"Fold AUC: {score:.4f}")

This gives you an honest performance estimate. If fold scores range from 0.52 to 0.58, your model has an edge. If they range from 0.45 to 0.70, it’s noise.

XGBoost Parameters That Fight Overfitting

Not all regularization is equal. These matter most for Nifty data:

1. max_depth — keep it shallow (3-5). Deep trees memorize intraday noise.
2. min_child_weight — higher values prevent single-sample splits.
3. gamma — requires a minimum loss reduction to split a node.
4. subsample + colsample_bytree — stochastic training reduces variance.
5. early_stopping_rounds — stop when validation AUC stops improving.

Example safer config:

model = xgb.XGBClassifier(
    n_estimators=300,
    max_depth=4,
    learning_rate=0.03,
    subsample=0.8,
    colsample_bytree=0.7,
    min_child_weight=5,
    gamma=0.1,
    early_stopping_rounds=30
)

Feature Engineering: Less Is More

New retail coders add 50+ features: RSI, MACD, SMA crossovers, option Greeks, news sentiment — all at once.

Problem: correlated features inflate importance scores and hide true drivers.

Better approach:

1. Start with 5-8 features (strike, IV, OI change, volume, days-to-expiry)
2. Check mutual information / correlation matrix
3. Drop features with >0.85 correlation
4. Re-train and compare WFV scores

The Expiry Calendar Effect

Nifty options expire every Thursday. The last two days before expiry show completely different dynamics (theta acceleration, pin-risk, max-pain clustering).

If you train without a “days_to_expiry” feature or without segmenting pre-expiry vs post-expiry, your model sees a blended signal that doesn’t generalize.

Fix: Add days_to_expiry as a feature, and consider training separate models for:

• Weekly expiry week (Mon-Wed)
• Expiry day (Thu)
• Post-expiry (Fri)

Optuna: Let the Data Choose Parameters

Manual tuning is biased toward what worked last month. Optuna automates the search and, more importantly, logs every trial.

import optuna

def objective(trial):
    params = {
        'max_depth': trial.suggest_int('max_depth', 3, 6),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.1, log=True),
        'subsample': trial.suggest_float('subsample', 0.6, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.6, 1.0),
    }
    model = xgb.XGBClassifier(n_estimators=300, **params)
    # cross-val inside Optuna
    scores = cross_val_score(model, X, y, cv=tscv, scoring='roc_auc')
    return scores.mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50)

On Android via Termux, use SQLite storage so trials survive reboots:

pip install optuna sqlite3
python train.py  # runs overnight, reviews at 9:15 AM

Data Leakage: The Silent Killer

Data leakage happens when information from the future sneaks into training. Examples in Nifty trading:

• Using today’s close to predict today’s intraday move
• Scaling the entire dataset before splitting
• Including expiry-day max-pain (computed after market close) in a pre-open model

Rule of thumb: if you can’t compute the feature at 9:15 AM on the day of the trade, it belongs in the test set — not the training set.

Honest Model Evaluation Metrics

AUC is popular but misleading for imbalanced datasets. Nifty moves are often small; most days “no trade” is the correct label.

Better metrics for retail trading models:

If your model has AUC 0.62 but Profit Factor 1.8, it’s a trading edge. If AUC is 0.92 but Profit Factor 0.7, it’s overfitted garbage.

Practical Checklist Before Going Live

1. WFV scores stable (std dev < 0.05 across folds)
2. Feature importance consistent across time chunks
3. No data leakage (can you compute features at trade time?)
4. Walk-forward paper trading for at least 2 expiry cycles
5. Position sizing based on Kelly or fixed-risk per trade
6. Stop-loss + max-drawdown guardrails hardcoded

Final Thought

Overfitting isn’t a math problem. It’s a honesty problem. The market doesn’t care about your training AUC. It cares whether you deployed capital responsibly.

Walk-forward validation keeps you honest. XGBoost is just a tool — the edge is in the process.

Shakti Tiwari
Nifty Option Trader · Research Analyst · XGBoost Expert
shaktitiwari715-ai.github.io/shakti-tiwari-nse

Tags

nifty #xgboost #machinelearning #overfitting #walkforward #python #termux #retailtrader #india #options