ML won't make you rich overnight, but it can add a measurable edge. We'll build a complete ML pipeline: feature engineering, model selection, time-series cross-validation, and ensemble methods — with realistic expectations.
Machine learning in trading is not magic. It's a tool for finding weak, noisy patterns in data that rule-based systems miss. The signal-to-noise ratio in financial data is extremely low, which means ML models overfit easily and degrade quickly.
| Approach | Works For | Doesn't Work For | Typical Edge |
|---|---|---|---|
| Classification | Direction prediction (up/down) | Magnitude prediction | 51-55% accuracy |
| Regression | Relative ranking of stocks | Exact return prediction | IC = 0.03-0.08 |
| Ensemble + Rules | Combining ML with technical rules | Replacing all rules | 10-20% Sharpe improvement |
| Deep Learning (LSTM) | Pattern recognition in sequences | Small datasets, interpretability | Marginal vs tree models |
Financial markets are an adversarial environment. When a pattern is discovered, money flows in to exploit it, which weakens or eliminates the pattern. This is why ML models degrade over time and need constant retraining. The edge from ML is not a 90% accuracy model — it's a 52% accuracy model that trades thousands of times with proper sizing and risk management. Small edges, applied consistently, compound into significant returns.
Features are the inputs to your ML model. In trading, features are derived from price, volume, and external data. The quality of your features determines the ceiling of your model's performance.
import pandas as pd import numpy as np import talib class FeatureEngine: """Generate ML features from OHLCV data.""" def build(self, data: pd.DataFrame) -> pd.DataFrame: """Build all features. Returns ~45 features.""" c = data['Close'] h = data['High'] l = data['Low'] v = data['Volume'] f = pd.DataFrame(index=data.index) # === Price returns (multiple lookbacks) === for lb in [1, 5, 10, 20, 60]: f[f'ret_{lb}'] = c.pct_change(lb) f[f'vol_{lb}'] = c.pct_change().rolling(lb).std() # === Normalized price position === for lb in [20, 60]: f[f'zscore_{lb}'] = (c - c.rolling(lb).mean()) / c.rolling(lb).std() f[f'pct_from_high_{lb}'] = c / c.rolling(lb).max() - 1 f[f'pct_from_low_{lb}'] = c / c.rolling(lb).min() - 1 # === Technical indicators === f['rsi_14'] = talib.RSI(c.values, 14) / 100 f['rsi_7'] = talib.RSI(c.values, 7) / 100 macd, sig, hist = talib.MACD(c.values) f['macd_hist'] = hist f['adx'] = talib.ADX(h.values, l.values, c.values, 14) / 100 f['cci'] = talib.CCI(h.values, l.values, c.values, 14) / 200 # === Volume features === f['vol_ratio_20'] = v / v.rolling(20).mean() f['obv_slope'] = (np.sign(c.diff()) * v).cumsum().pct_change(10) f['mfi'] = talib.MFI(h.values, l.values, c.values, v.values, 14) / 100 # === Candlestick patterns (binary) === f['body_ratio'] = (c - data['Open']).abs() / (h - l + 1e-8) f['upper_shadow'] = (h - np.maximum(c, data['Open'])) / (h - l + 1e-8) # === Calendar features === f['dow'] = data.index.dayofweek / 4 f['month'] = data.index.month / 12 f['month_end'] = (data.index.day > 25).astype(float) return f.dropna() def create_labels(self, close: pd.Series, horizon: int = 5, method: str = 'binary') -> pd.Series: """Create prediction labels. binary: +1 if forward return > 0, else 0 triple: +1 (up > threshold), 0 (flat), -1 (down) """ fwd_ret = close.pct_change(horizon).shift(-horizon) if method == 'binary': return (fwd_ret > 0).astype(int) elif method == 'triple': threshold = fwd_ret.std() * 0.5 labels = pd.Series(0, index=close.index) labels[fwd_ret > threshold] = 1 labels[fwd_ret < -threshold] = -1 return labels
1) No future data: every feature at time t must use only data from t-1 and earlier. 2) Normalize: scale all features to similar ranges (z-score or [0,1]). 3) Remove correlated features: if correlation > 0.9, drop one. 4) Domain knowledge: RSI, MACD, ATR are better features than raw prices because they capture market microstructure. 5) Labels: use forward returns, not contemporaneous returns (data leakage!).
from sklearn.ensemble import RandomForestClassifier from sklearn.preprocessing import StandardScaler from sklearn.pipeline import Pipeline import xgboost as xgb class TradingModels: """ML models for trading signals.""" @staticmethod def random_forest() -> Pipeline: """Random Forest — robust, interpretable, hard to overfit. The workhorse of quant ML.""" return Pipeline([ ('scaler', StandardScaler()), ('model', RandomForestClassifier( n_estimators=500, max_depth=5, # Shallow trees = less overfit min_samples_leaf=50, # Require 50+ samples per leaf max_features='sqrt', # Random feature subset class_weight='balanced', random_state=42, n_jobs=-1 )) ]) @staticmethod def xgboost_model() -> Pipeline: """XGBoost — gradient boosting, often best on tabular data. More powerful but more prone to overfitting.""" return Pipeline([ ('scaler', StandardScaler()), ('model', xgb.XGBClassifier( n_estimators=200, max_depth=4, learning_rate=0.05, subsample=0.8, colsample_bytree=0.8, reg_alpha=0.1, # L1 regularization reg_lambda=1.0, # L2 regularization min_child_weight=50, eval_metric='logloss', random_state=42, use_label_encoder=False )) ])
| Model | Overfit Risk | Interpretability | Speed | Typical OOS Accuracy |
|---|---|---|---|---|
| Random Forest | Low | High (feature importance) | Fast | 51-53% |
| XGBoost | Medium | Medium | Fast | 52-54% |
| LSTM | High | Low (black box) | Slow | 51-53% |
| Logistic Regression | Very Low | Very High | Fastest | 50-52% |
For daily-frequency trading with hundreds of features and a few thousand samples, tree-based models (Random Forest, XGBoost) consistently outperform deep learning. LSTMs and Transformers shine with enormous datasets (tick data, order book data, millions of samples) and complex temporal patterns. For our use case, XGBoost is king. Don't use a cannon to kill a mosquito.
Standard K-fold CV is WRONG for time series. It shuffles data, creating future-leakage. Purged K-fold respects temporal ordering and adds a gap between train and test sets to prevent information leakage from overlapping labels.
class PurgedKFold: """ Purged K-Fold Cross-Validation for time series. 1. Split data into K chronological folds 2. Train on folds [0..i-1], test on fold [i] 3. Add a purge gap between train/test to prevent leakage """ def __init__(self, n_splits: int = 5, purge_gap: int = 10): self.n_splits = n_splits self.purge_gap = purge_gap def split(self, X: pd.DataFrame): """Generate train/test indices.""" n = len(X) fold_size = n // (self.n_splits + 1) for i in range(1, self.n_splits + 1): train_end = fold_size * i test_start = train_end + self.purge_gap test_end = test_start + fold_size if test_end > n: break train_idx = np.arange(0, train_end) test_idx = np.arange(test_start, test_end) yield train_idx, test_idx def evaluate_model(model, X, y, cv=None): """Evaluate model with purged K-fold. Returns OOS metrics, not in-sample.""" if cv is None: cv = PurgedKFold(n_splits=5, purge_gap=10) results = [] for train_idx, test_idx in cv.split(X): X_train = X.iloc[train_idx] y_train = y.iloc[train_idx] X_test = X.iloc[test_idx] y_test = y.iloc[test_idx] model.fit(X_train, y_train) preds = model.predict(X_test) proba = model.predict_proba(X_test)[:, 1] # Trading-relevant metrics (NOT accuracy) fwd_returns = y_test # Actual forward returns long_mask = preds == 1 short_mask = preds == 0 results.append({ 'accuracy': (preds == y_test.values).mean(), 'long_return': fwd_returns[long_mask].mean() if long_mask.any() else 0, 'n_test': len(test_idx), 'n_long': long_mask.sum(), }) return pd.DataFrame(results)
A model with 55% accuracy that is correct on small moves and wrong on big moves will lose money. A model with 48% accuracy that catches big moves and avoids big losses can be very profitable. Always evaluate ML trading models using Sharpe ratio, profit factor, and max drawdown of the resulting portfolio — never classification accuracy alone.
The most practical approach: use ML to augment, not replace, your rule-based strategy. The ensemble combines the interpretability and robustness of rules with the pattern-detection ability of ML.
class HybridStrategy: """Combine ML predictions with rule-based signals. The ML model acts as a filter or confidence booster for the rule-based signals, not a replacement. """ def __init__(self, ml_model, rule_strategy, ml_weight: float = 0.4, rule_weight: float = 0.6, threshold: float = 0.3): self.ml = ml_model self.rules = rule_strategy self.ml_w = ml_weight self.rule_w = rule_weight self.threshold = threshold def generate_signals(self, data: pd.DataFrame, features: pd.DataFrame) -> pd.Series: """Generate ensemble signals.""" # Rule-based signal (from Part 3) rule_signal = self.rules.generate_signals(data) # ML probability (0 to 1, where > 0.5 = bullish) ml_proba = self.ml.predict_proba(features)[:, 1] ml_signal = pd.Series(ml_proba * 2 - 1, # Map to [-1, 1] index=features.index) # Weighted ensemble ensemble = self.rule_w * rule_signal + self.ml_w * ml_signal # Convert to discrete signals signals = pd.Series(0, index=data.index) signals[ensemble > self.threshold] = 1 signals[ensemble < -self.threshold] = -1 return signals def retrain(self, features, labels, lookback: int = 504): """Retrain ML model on recent data. Call monthly or when performance degrades.""" recent_X = features.tail(lookback) recent_y = labels.tail(lookback) self.ml.fit(recent_X, recent_y) print(f"Model retrained on {len(recent_X)} samples")
1) Predicting exact prices. Markets are not deterministic. 2) Training on the full dataset and reporting in-sample results. Always evaluate OOS. 3) 100+ features with 2 years of data. Massive overfitting. 4) Deep learning on daily bars. Not enough data; use tree models instead. 5) Ignoring transaction costs. A model that trades 50x per day needs an enormous edge to be profitable. 6) No retraining schedule. Markets evolve; your model must too.
A well-built ML trading system might achieve: OOS accuracy: 52-55% (not 90%). Information Coefficient: 0.03-0.08 (not 0.5). Sharpe improvement: 10-20% over the rule-based baseline (not 10x). Model half-life: 3-6 months before significant degradation. These are small numbers, but small edges applied consistently with proper risk management compound into real profits. The goal is not a crystal ball — it's a slightly loaded coin flipped thousands of times.
# The complete quant trading system we've built: # # Part 1: DATA # yfinance -> raw OHLCV -> cleaned DataFrame # # Part 2: BACKTEST # BacktestEngine -> metrics (Sharpe, drawdown, PF) # # Part 3: SIGNALS # Momentum + Reversion + Volume + Volatility # -> CompositeSignal -> weighted score # # Part 4: RISK # PositionSizer (Kelly) + StopLoss (ATR) # + DrawdownController + PortfolioOptimizer # # Part 5: LIVE # Alpaca API -> ExecutionEngine -> TradeJournal # Paper trading -> 3-phase deployment # # Part 6: ML (this part) # FeatureEngine -> XGBoost/RF -> PurgedKFold # -> HybridStrategy (ML + Rules) # # === THE DAILY LOOP === # 1. Fetch latest data (yfinance / Alpaca) # 2. Generate features (FeatureEngine) # 3. ML prediction (XGBoost probability) # 4. Rule-based signals (CompositeSignal) # 5. Ensemble (HybridStrategy) # 6. Risk check (DrawdownController) # 7. Position sizing (Kelly/ATR) # 8. Execute (Alpaca bracket orders) # 9. Log (TradeJournal) # 10. Monitor (LiveMonitor dashboard)