Ensemble methods – Random Forest & Gradient Boosting

Subject: Data Science & Artificial Intelligence
School level: 12th grade – HTL Informatik
Requirements: Supervised Learning, Decision Trees, Python / scikit-learn
Author: HTL Pinkafeld – IF/IT

1. Why ensemble methods?

A single decision tree has a known problem: it is highly prone to overfitting - the tree memorizes the training data but generalizes poorly to new data.

The idea behind ensembles

“Many weak learners together make one strong learner.”

Instead of a single complex model, ensemble methods combine many simple models (so-called base learners) into a common prediction. The analogy from everyday life:

• Individual expert decides → risky, depending on form on the day
• Committee of experts decides → Individual errors are compensated for

Advantages over individual models

2. Basic principles: Bagging and boosting

There are two basic strategies for how models can be combined:

2.1 Bagging (Bootstrap Aggregating)

Bagging involves training multiple models independently on different subsets of the training data. The predictions are then aggregated.

Training data
│
├─ Random sample 1 → Model 1 ─┐
├─ Random sample 2 → Model 2 ─┤→ Majority vote / mean
├─ Random sample 3 → Model 3 ─┤
└─ ... ┘

• Samples with reset (bootstrap): a data point can occur multiple times
• Models are trained in parallel
• Goal: Reduce variance

2.2 Boosting

Boosting trains models sequentially. Each new model focuses on the flaws of its predecessor.

Training data
│
└─ Model 1 → Analyze errors
│
Model 2 (focused on errors of M1) → Analyze errors
│
Model 3 (focused on errors of M2)
│
Final prediction (weighted total)

• Models are trained sequentially
• Each model corrects the residual error of its predecessor
• Goal: Reduce bias

2.3 Comparison

3. Random Forest

3.1 Basic idea

Random Forest is the most famous bagging method. Many decision trees are trained, which differ in two points:

1. Random data sampling: Each tree is trained on a bootstrap sample (~63% of the data)
2. Random feature selection: Each split only considers a random subset of features

The randomness ensures that the trees are diversified - they make different errors that cancel each other out when aggregated.

3.2 Prediction

• Classification: Majority voting – the class with the most votes wins
• Regression: Average of all tree predictions

3.3 Out of Bag Error (OOB)

Since each tree is only trained on ~63% of the data, the remaining ~37% (out-of-bag samples) can be used as a validation set - without a separate test split.

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load and split data
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

# Train random forest
rf = RandomForestClassifier(
    n_estimators=100,    # Number of trees
    max_depth=None,      # Trees grow fully
    max_features='sqrt', # √(Anzahl Features) pro Split
    oob_score=True,      # Out-of-Bag-Evaluation aktivieren
    random_state=42
)
rf.fit(X_train, y_train)

# Results
print(f"Test Accuracy:  {accuracy_score(y_test, rf.predict(X_test)):.2%}")
print(f"OOB Accuracy:   {rf.oob_score_:.2%}")

3.4 Important hyperparameters

3.5 Strengths and Weaknesses

4. Gradient boosting

4.1 Basic idea

Gradient boosting builds trees sequentially, with each new tree minimizing the residual error of the previous model.

The name is derived from gradient descent optimization: instead of optimizing parameters of a single model, new models are added in the direction of the greatest error reduction.

4.2 Algorithm step by step

1. Start with a simple model F₀ (e.g. mean of y)

2. For each iteration t = 1, 2, ..., T:
a) Calculate the residuals: rᵢ = yᵢ - Fₜ₋₁(xᵢ)
b) Train a new tree hₜ on the residuals
c) Update the model:
Fₜ(x) = Fₜ₋₁(x) + η hₜ(x)
(η = learning rate, controls step size)

3. Final prediction: F_T(x)

Intuition: Each new tree “repairs” the errors of all previous trees. The learning rate η controls how much each tree contributes - small values ​​lead to more robust models, but need more trees.

4.3 Scikit-learn: Gradient boosting

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report

# Breast cancer dataset (binary classification)
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# Gradient boosting model
gb = GradientBoostingClassifier(
    n_estimators=200,      # Anzahl Boosting-Runden
    learning_rate=0.1,     # Lernrate η
    max_depth=3,           # Shallow trees are typical!
    subsample=0.8,         # Stochastic Gradient Boosting
    random_state=42
)
gb.fit(X_train, y_train)

print(classification_report(y_test, gb.predict(X_test),
      target_names=data.target_names))

4.4 Important hyperparameters

Important rule of thumb: learning_rate and n_estimators are opposite - small learning rate needs more trees. Typical combination: learning_rate=0.05, n_estimators=500.

5. XGBoost – Extreme Gradient Boosting

5.1 What is XGBoost?

XGBoost is a highly optimized implementation of gradient boosting that has set the benchmark in many Kaggle competitions. Compared to scikit-learn's GradientBoosting it offers:

• Speed: Parallelization and cache-optimized data structures
• Regularization: Built-in L1 and L2 regularization
• Missing values: Automatic handling without preprocessing
• Early stop: Training stops automatically when there is no further improvement

5.2 Installation and Use

pip install xgboost

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# XGBoost model (scikit-learn API)
model = xgb.XGBClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=4,
    subsample=0.8,
    colsample_bytree=0.8,   # Anteil Features pro Baum
    reg_alpha=0.1,           # L1-Regularisierung
    reg_lambda=1.0,          # L2-Regularisierung
    use_label_encoder=False,
    eval_metric='logloss',
    random_state=42
)

# Training with early stopping
model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    early_stopping_rounds=20,  # Stoppt wenn 20 Runden keine Verbesserung
    verbose=False
)

print(f"Beste Iteration: {model.best_iteration}")
print(f"Test Accuracy: {accuracy_score(y_test, model.predict(X_test)):.2%}")

5.3 Early Stopping

Early Stopping automatically prevents overfitting:

Iteration 1: Validation Loss = 0.342
Iteration 5: Validation Loss = 0.198
Iteration 10: Validation Loss = 0.145 ← Minimum
Iteration 15: Validation Loss = 0.148
Iteration 20: Validation Loss = 0.151
Iteration 30: Validation Loss = 0.163 ← Stop! (20 rounds without improvement)

6. Comparison of ensemble methods

6.1 Direct comparison on breast cancer data set

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer
import xgboost as xgb
import numpy as np

data = load_breast_cancer()
X, y = data.data, data.target

modelle = {
    'Decision Tree':      DecisionTreeClassifier(max_depth=5, random_state=42),
    'Random Forest':      RandomForestClassifier(n_estimators=100, random_state=42),
    'Gradient Boosting':  GradientBoostingClassifier(n_estimators=100, random_state=42),
    'XGBoost':            xgb.XGBClassifier(n_estimators=100, random_state=42,
                              eval_metric='logloss', verbosity=0),
}

print(f"{'Model':<22} {'CV Accuracy':>12} {'Std':>8}")
print("-" * 45)
for name, model in modelle.items():
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    print(f"{name:<22} {scores.mean():>11.2%} {scores.std():>7.2%}")

6.2 When which method?

7. Hyperparameter tuning

7.1 Grid Search

Systematic search of all parameter combinations:

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()

param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth':    [None, 5, 10],
    'max_features': ['sqrt', 'log2'],
}

rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(
    rf, param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1,       # alle CPU-Kerne nutzen
    verbose=1
)
grid_search.fit(data.data, data.target)

print(f"Beste Parameter: {grid_search.best_params_}")
print(f"Beste CV Accuracy: {grid_search.best_score_:.2%}")

7.2 Random Search

For many hyperparameters, Random Search is more efficient than Grid Search - random combinations are tested:

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'n_estimators': randint(50, 500),
    'max_depth':    [None, 3, 5, 10, 20],
    'max_features': ['sqrt', 'log2', 0.3, 0.5],
    'min_samples_leaf': randint(1, 10),
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_dist,
    n_iter=50,       # test 50 random combinations
    cv=5,
    scoring='accuracy',
    n_jobs=-1,
    random_state=42
)
random_search.fit(data.data, data.target)
print(f"Beste Parameter: {random_search.best_params_}")

7.3 Comparison of search methods

8. Feature Importance and Interpretability

8.1 Feature Importance from Random Forest

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(data.data, data.target)

# Feature Importance as a DataFrame
importance_df = pd.DataFrame({
    'feature': data.feature_names,
    'importance': rf.feature_importances_
}).sort_values('importance', ascending=True).tail(10)

# Visualization
importance_df.plot(kind='barh', x='feature', y='importance', legend=False)
plt.xlabel('Wichtigkeit (Gini-Impurity Reduktion)')
plt.title('Top 10 wichtigste Features – Random Forest')
plt.tight_layout()
plt.show()

8.2 Permutation Importance

A more robust alternative: Features are randomly shuffled and the drop in model quality is measured.

from sklearn.inspection import permutation_importance

result = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)

perm_df = pd.DataFrame({
    'feature': data.feature_names,
    'importance_mean': result.importances_mean,
    'importance_std':  result.importances_std
}).sort_values('importance_mean', ascending=False).head(10)

print(perm_df.to_string(index=False))

8.3 SHAP values ​​(outlook)

For deeper interpretability, the SHAP library offers explainable predictions at a case-by-case level:

# pip install shap
import shap

explainer = shap.TreeExplainer(rf)
shap_values = explainer.shap_values(X_test[:10])

# Visualization for a single data point
shap.force_plot(explainer.expected_value[1], shap_values[1][0], X_test[0],
                feature_names=data.feature_names)

9. Practical Project: Titanic Survival

9.1 Task

Based on passenger data (age, class, gender, ...) it should be predicted whether a person survived the Titanic sinking.

9.2 Full pipeline

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import classification_report, confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt

# ── 1. Create data (simplified) ──────────────────────────────────────
np.random.seed(42)
n = 500

data = pd.DataFrame({
    'pclass':    np.random.choice([1, 2, 3], n, p=[0.25, 0.25, 0.5]),
    'sex':       np.random.choice(['male', 'female'], n, p=[0.65, 0.35]),
    'age':       np.random.normal(30, 14, n).clip(1, 80),
    'sibsp':     np.random.choice([0, 1, 2, 3], n, p=[0.6, 0.25, 0.1, 0.05]),
    'fare':      np.random.exponential(35, n).clip(5, 500),
    'embarked':  np.random.choice(['S', 'C', 'Q'], n, p=[0.72, 0.19, 0.09]),
})

# Survival depends on class and gender
survive_prob = (
    (data['sex'] == 'female') * 0.5 +
    (data['pclass'] == 1) * 0.2 +
    (data['pclass'] == 2) * 0.1 -
    (data['age'] > 60) * 0.1 + 0.15
).clip(0.05, 0.95)
data['survived'] = np.random.binomial(1, survive_prob)

# ── 2. Preprocessing ─────────────────────────── ───────────────────────────
le = LabelEncoder()
data['sex_enc'] = le.fit_transform(data['sex'])
data['embarked_enc'] = le.fit_transform(data['embarked'])

features = ['pclass', 'sex_enc', 'age', 'sibsp', 'fare', 'embarked_enc']
X = data[features]
y = data['survived']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# ── 3. Compare models ──────────────────────── ────────────────────────
modelle = {
    'Random Forest':     RandomForestClassifier(n_estimators=100, random_state=42),
    'Gradient Boosting': GradientBoostingClassifier(n_estimators=100, learning_rate=0.1,
                                                     max_depth=3, random_state=42),
}

for name, model in modelle.items():
    model.fit(X_train, y_train)
    cv = cross_val_score(model, X, y, cv=5).mean()
    print(f"\n{name}  (CV: {cv:.2%})")
    print(classification_report(y_test, model.predict(X_test),
                                  target_names=['Did not survive', 'Survived']))

# ── 4. Visualize Confusion Matrix ────────────────────────────────────
best_model = modelle['Random Forest']
cm = confusion_matrix(y_test, best_model.predict(X_test))
sns.heatmap(cm, annot=True, fmt='d', cmap='Oranges',
            xticklabels=['Did not survive', 'Survived'],
            yticklabels=['Did not survive', 'Survived'])
plt.title('Confusion Matrix – Random Forest Titanic')
plt.tight_layout()
plt.show()

10. Summary and outlook

Overview of ensemble methods

Checklist: Use ensemble methods

✅ Starting point: Random Forest as a fast, robust baseline
✅ For maximum accuracy: XGBoost / Gradient Boosting
✅ Hyperparameters: n_estimators + learning_rate tune first
✅ Check overfitting: train score vs. test score / cross-validation
✅ Analyze feature importance: remove irrelevant features
✅ Enable early stopping for boosting methods

Outlook: Next topics

• Unsupervised Learning: K-Means Clustering, PCA, dimensionality reduction
• Deep Learning: Neural Networks, Backpropagation, CNNs
• Natural Language Processing: Text classification, transformer models
• MLOps: Version, deploy, and monitor models

Appendix: Quick Reference Ensemble Methods

# Random forest
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
rf = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)

# Gradient boosting
from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor
gb = GradientBoostingClassifier(n_estimators=200, learning_rate=0.05, max_depth=3)

# XGBoost
import xgboost as xgb
xgb_model = xgb.XGBClassifier(n_estimators=300, learning_rate=0.05,
                                max_depth=4, subsample=0.8,
                                colsample_bytree=0.8, eval_metric='logloss')

# Hyperparameter tuning
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Feature Importance
importances = rf.feature_importances_          # Gini-basiert
from sklearn.inspection import permutation_importance
perm = permutation_importance(rf, X_test, y_test, n_repeats=10)

HTL Pinkafeld – IF/IT | Data Science & AI | 12th grade