×
🌳 XGBoost Overview – The King of Machine Learning Algorithms for Quantitative Finance
What is XGBoost?
XGBoost stands for eXtreme Gradient Boosting. "Gradient boosting" is the technique of repeatedly correcting errors, and "extreme" refers to all the clever optimizations that make it blazing fast and highly accurate – even on gigantic datasets.
Think of XGBoost as a super‑charged committee of decision trees. A single decision tree is like a flowchart: "Is the price yesterday > $100? → go left, else right." One tree is weak – it makes lots of mistakes. XGBoost builds trees one after another, and each new tree is trained to fix the errors of all previous trees combined. Over 100 or 1000 rounds, the committee becomes incredibly sharp, learning complex, non‑linear patterns that no single tree could ever capture.
✅ Pros – Why quants love it
- Handles missing data automatically. You can have gaps in your data (IPOs, delistings) and XGBoost learns the best direction to send those missing values – no need to drop or impute rows.
- Throwing everything in works. You can include hundreds of predictors, many of which may be noise. XGBoost is excellent at identifying the few that truly matter and ignoring the rest. The Top Feature Importance chart you see after every forecast is proof of that.
- Built‑in regularisation. Parameters like alpha and lambda prevent the model from memorising random noise, so it generalises better to unseen data.
- Speed. It's highly optimised – training finishes in seconds, not hours.
- Industry standard. XGBoost dominates Kaggle competitions, quantitative finance, fraud detection, and any tabular‑data problem you can think of.
⚠️ Cons – Know the trade‑offs
- Can overfit. Too many trees or too deep a tree, and the model starts learning noise instead of signal. That's why we provide hyperparameter controls.
- Needs tuning. Default settings are good, but the best performance comes from careful tweaking (or using our Fine‑Tune button!).
- Less interpretable than a linear model. You can't just read a single coefficient – but the feature importance output helps you understand which predictors drive the forecast.
⚙️ Hyperparameters – Your Control Panel
Each of these knobs influences how the model learns. Tweak them to improve your forecast – or let Fine‑Tune search for the best combination.
- Learning Rate (eta) [Range: 0.001 – 0.99, typical 0.01–0.3]
- How big a step each tree takes toward fixing errors. A smaller value (e.g. 0.01) makes the model more robust but requires more boosting rounds. A larger value (e.g. 0.3) learns faster but risks overshooting.
- Max Depth [Range: 1 – no limit, typical 3–10]
- The maximum depth of a single tree. Deeper trees capture more complex interactions but can overfit. Values of 3–10 are common. Our default is 10.
- Min Child Weight [Range: 0 – ∞, typical 1–10]
- The minimum sum of instance weights (or hessian) needed in a child node. Increasing it forces the model to be more conservative, avoiding splits that only explain a handful of outliers. Higher values → less overfitting.
- Subsample [Range: 0.1 – 1.0]
- The fraction of training rows randomly sampled for each tree. A value of 0.6 means each tree sees 60% of the data, which adds randomness and reduces overfitting. 1.0 uses the whole dataset.
- Colsample Bytree [Range: 0.1 – 1.0]
- Fraction of features randomly sampled for each tree. 0.9 means each tree sees 90% of the available features.
- Colsample Bylevel [Range: 0.1 – 1.0]
- Fraction of features sampled at each depth level. Using this adds another layer of randomness.
- Colsample Bynode [Range: 0.1 – 1.0]
- Fraction of features sampled for every split (node). Combining all three column‑sampling parameters gives the strongest regularisation.
- Gamma (min_split_loss) [Range: 0 – ∞, typical 0–5]
- Minimum loss reduction required to make a further partition on a leaf node. Think of it as a "hurdle" – the split must improve the model by at least this much or it won't happen. Higher gamma → fewer splits → simpler model.
- Alpha (L1 regularisation) [Range: 0 – ∞, typical 0–10]
- Penalty on the absolute magnitude of leaf weights. Encourages sparsity – many leaves may end up with zero contribution, effectively removing unimportant branches.
- Lambda (L2 regularisation) [Range: 0 – ∞, typical 0–10]
- Penalty on the squared magnitude of leaf weights. Smooths the model and prevents any single leaf from having an overly large influence.
- Max Leaves [Range: 0 – no limit, typical 0–512]
- The maximum number of terminal nodes in each tree. When grow_policy is "lossguide", this is the main way to control tree complexity. Higher values allow finer‑grained splits.
- Grow Policy [Options: depthwise, lossguide]
- Depthwise grows trees level by level (more balanced), while lossguide grows the leaf with the highest loss change (can produce asymmetric trees). Lossguide often yields better accuracy but may overfit if max_leaves is too high.
- Tree Method [Options: hist, approx, exact, gpu_hist – we use hist]
- The algorithm used to construct trees. hist is the standard fast histogram‑based method; it's what we use. GPU options appear if a compatible GPU is available.
- Num Boost Round [Range: 1 – ∞, typical 100–2000]
- The number of trees to build. More trees mean the model can learn more complex patterns, but also increase training time and the risk of overfitting. Early stopping (used in fine‑tuning) can automatically pick the optimal number.
📉 Loss Functions – What the Model Tries to Minimize
The loss function measures how "wrong" a prediction is, and the model uses that feedback to improve. We offer two choices:
- reg:squarederror – The classic mean squared error. Penalizes large errors heavily. Best when you believe outliers are genuinely informative and should influence the forecast strongly.
- Custom (Smooth Pseudo‑Huber Loss) – A smooth approximation of Huber loss. For small residuals it behaves like squared error; as residuals grow it smoothly transitions to a nearly linear penalty, making the model robust to outliers while keeping the objective fully differentiable. The Delta parameter sets the scale of this transition. You can optionally weight each day's contribution by the Weight Column (typically a dollar‑volume column) so that high‑volume days carry more influence.
🧠 How Your Forecast Comes Together
When you hit Forecast, here's what happens behind the scenes:
- Feature engineering. Your chosen predictors are combined with indicator periods you have customized (lags, EMAs, RSIs, volatility, FOMO) to create a rich feature matrix. Additionally, we have added time and volume features based on your Target Ticker and Predictors for you to enhance your model's learning capabilities.
- Training. The XGBoost model is trained on this feature matrix, using your selected hyperparameters and loss function, to predict the next day's value of the target ticker.
- Recursive forecasting. Once the model can predict one day ahead, we feed that prediction back as if it were a real observation, recompute all indicators, and predict the following day. This repeats for as many periods as you requested – the model literally "walks" into the future.
- Output. You see the historical data, the fitted (in‑sample) values, and the future forecast on the chart, along with the Top Feature Importance that tells you which predictors the model relied on most.
Every part of the model – from the predictors, the indicator periods, the hyperparameters, and even the loss function – directly impacts forecast quality.
To maximize forecast quality, we invite you to explore VIP membership with us to take full advantage of our rich database of predictors and state‑of‑the‑art fine‑tuning capabilities. We only ask for a small fee in return as training and finetuning the Models we provide for you can get quite expensive, computationally.
Thank you and if you have any questions, please feel free to email us at any time at team@stockpriceforecast.ai
XGBoost – making trees work for you since 2014 🌳