Portfolio Optimization in Python: From Markowitz to Machine Learning
Every quant faces the same question: how do you allocate capital across assets to maximize returns while minimizing risk? This is portfolio optimization — and it's the backbone of modern quantitative finance.
In this post, we'll build portfolio optimization from scratch in Python, starting with Markowitz's classical mean-variance framework and progressing to machine learning approaches used on actual trading desks.
The Markowitz Framework
Harry Markowitz's 1952 paper revolutionized finance by formalizing diversification. The core idea: don't just pick individual stocks — optimize the entire portfolio.
The Math Behind It
Given a portfolio of $n$ assets with:
- Expected return vector: $\mu$
- Covariance matrix: $\Sigma$
- Weight vector: $w$
The portfolio return and risk are:
The optimization problem becomes:
Python Implementation
import numpy as np
import pandas as pd
from scipy.optimize import minimize
def portfolio_stats(weights, mean_returns, cov_matrix):
"""Calculate portfolio return and volatility."""
portfolio_return = np.sum(mean_returns * weights)
portfolio_volatility = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights)))
return portfolio_return, portfolio_volatility
def minimize_volatility(target_return, mean_returns, cov_matrix, n_assets):
"""Find minimum volatility portfolio for a target return."""
args = (mean_returns, cov_matrix)
constraints = (
{'type': 'eq', 'fun': lambda x: np.sum(x) - 1},
{'type': 'eq', 'fun': lambda x: portfolio_stats(x, mean_returns, cov_matrix)[0] - target_return}
)
bounds = tuple((0, 1) for _ in range(n_assets))
result = minimize(
fun=lambda x: portfolio_stats(x, mean_returns, cov_matrix)[1],
x0=np.array([1/n_assets] * n_assets),
method='SLSQP',
bounds=bounds,
constraints=constraints
)
return result
# Generate sample data
np.random.seed(42)
n_assets = 5
tickers = ['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA']
# Simulate returns
mean_returns = np.array([0.12, 0.10, 0.15, 0.11, 0.20])
cov_matrix = np.array([
[0.04, 0.006, 0.008, 0.005, 0.01],
[0.006, 0.03, 0.007, 0.004, 0.008],
[0.008, 0.007, 0.06, 0.009, 0.015],
[0.005, 0.004, 0.009, 0.035, 0.007],
[0.01, 0.008, 0.015, 0.007, 0.10]
])
# Find efficient frontier
target_returns = np.linspace(0.08, 0.20, 50)
frontier_volatilities = []
for target in target_returns:
result = minimize_volatility(target, mean_returns, cov_matrix, n_assets)
frontier_volatilities.append(result.fun)
print("Efficient frontier computed!")
print(f"Minimum volatility: {min(frontier_volatilities):.4f}")
The Efficient Frontier
The efficient frontier is the set of portfolios offering the highest expected return for each level of risk. Any portfolio below the frontier is suboptimal — you could get more return for the same risk.
| Portfolio | Return | Volatility | Sharpe Ratio |
|---|---|---|---|
| Min Variance | 10.2% | 14.8% | 0.42 |
| Max Sharpe | 14.5% | 18.2% | 0.63 |
| Equal Weight | 13.6% | 19.1% | 0.50 |
Beyond Markowitz: Black-Litterman
Classical Markowitz has a problem: it's extremely sensitive to input estimates. Small changes in expected returns lead to wildly different allocations. The Black-Litterman model fixes this by combining market equilibrium with investor views.
The Key Insight
Instead of estimating returns directly, Black-Litterman starts with implied equilibrium returns derived from market capitalization weights:
where $\delta$ is the risk aversion coefficient and $w_{mkt}$ is the market portfolio.
Python Implementation
def black_litterman(cov_matrix, market_weights, risk_aversion, views, view_confidences):
"""
Black-Litterman model implementation.
Parameters:
- cov_matrix: covariance matrix
- market_weights: market cap weights
- risk_aversion: risk aversion coefficient
- views: matrix of investor views (k x n)
- view_confidences: confidence in each view (k x k)
"""
n = len(market_weights)
# Implied equilibrium returns
pi = risk_aversion * cov_matrix @ market_views
# View uncertainty (Omega)
tau = 0.05
sigma_views = view_confidences
# Black-Litterman expected returns
P = views
Q = np.array([0.10, 0.15]) # View returns
M1 = np.linalg.inv(tau * cov_matrix)
M2 = P.T @ np.linalg.inv(sigma_views) @ P
bl_returns = np.linalg.inv(M1 + M2) @ (M1 @ pi + P.T @ np.linalg.inv(sigma_views) @ Q)
return bl_returns
Machine Learning Approaches
Modern quant desks use ML for portfolio optimization in several ways:
1. Reinforcement Learning for Dynamic Allocation
import torch
import torch.nn as nn
class PortfolioAllocator(nn.Module):
def __init__(self, n_assets, lookback=60):
super().__init__()
self.lstm = nn.LSTM(n_assets, 64, batch_first=True)
self.fc = nn.Linear(64, n_assets)
self.softmax = nn.Softmax(dim=-1)
def forward(self, returns_history):
# returns_history: (batch, lookback, n_assets)
lstm_out, _ = self.lstm(returns_history)
weights = self.fc(lstm_out[:, -1, :])
return self.softmax(weights)
2. Variational Autoencoders for Scenario Generation
Instead of assuming normal returns, VAEs learn the actual distribution of market scenarios, capturing fat tails and regime changes.
3. Graph Neural Networks for Sector Allocation
GNNs model relationships between assets, capturing contagion effects and sector rotations that traditional correlation matrices miss.
Practical Considerations
Transaction Costs
Real portfolios face trading costs. The optimization must account for:
def portfolio_with_transaction_costs(weights, new_weights, transaction_cost=0.001):
"""Add transaction cost penalty."""
turnover = np.sum(np.abs(new_weights - weights))
cost = turnover * transaction_cost
return cost
Rebalancing Frequency
| Frequency | Pros | Cons |
|---|---|---|
| Daily | Captures signals | High transaction costs |
| Weekly | Balance | Moderate costs |
| Monthly | Low costs | May miss opportunities |
Risk Constraints
# Maximum position size
constraints.append({'type': 'ineq', 'fun': lambda x: 0.20 - x})
# Maximum sector exposure
sector_exposure = np.sum(weights[sector_indices])
constraints.append({'type': 'ineq', 'fun': lambda x: 0.40 - sector_exposure})
Real-World Application
Here's a complete pipeline you can use:
import yfinance as yf
from sklearn.covariance import LedoitWolf
# Fetch data
data = yf.download(['AAPL', 'MSFT', 'GOOGL', 'AMZN', 'TSLA'],
start='2020-01-01', end='2024-01-01')
returns = data['Adj Close'].pct_change().dropna()
# Shrinkage estimator for covariance
lw = LedoitWolf().fit(returns)
cov_matrix = lw.covariance_
mean_returns = returns.mean() * 252 # Annualized
# Optimize
result = minimize_volatility(0.12, mean_returns, cov_matrix, 5)
print(f"Optimal weights: {dict(zip(tickers, result.x.round(3)))}")
Key Takeaways
- Markowitz is the foundation — understand mean-variance optimization before moving to advanced methods
- Black-Litterman fixes sensitivity — use market equilibrium as a prior
- ML adds value — but only when you have enough data and proper validation
- Transaction costs matter — theoretical optimal ≠ practical optimal
- Risk constraints are essential — never optimize without limits
Want to Learn More?
If you're preparing for quant interviews or want to deepen your portfolio theory knowledge, check out our Quantitative Finance for Absolute Beginners course — it covers portfolio optimization, risk management, and derivatives pricing with hands-on Python projects.
Built with Python, NumPy, SciPy, and PyTorch. Full code available on GitHub.
