Unsupervised learning – clustering and dimensionality reduction

Subject: Data Science & Artificial Intelligence
School level: 12th grade – HTL Informatik
Requirements: Supervised Learning, Python / scikit-learn, NumPy, Matplotlib
Author: HTL Pinkafeld – IF/IT

1. What is Unsupervised Learning?

In supervised learning, we have a label for each data point - a known answer that the model is trained towards. These labels do not exist in Unsupervised Learning. The algorithm only receives the input data and is supposed to independently find structures, patterns or groupings in it.

Comparison of learning paradigms

Typical use cases

• Customer segmentation: Divide customers into groups based on purchasing behavior
• Anomaly Detection: Find unusual transactions (credit card fraud).
• Document grouping: Cluster news articles thematically
• Recommendation systems: Identify similar user profiles
• Data Compression: Simplify images or signals without significant loss of information
• Visualization: Display high-dimensional data in 2D/3D

Main areas

Unsupervised learning
│
├── Clustering
│ ├── K-Means
│ ├── DBSCAN
│ └── Hierarchical clustering
│
└── Dimensional reduction
├── PCA (Principal Component Analysis)
└── t-SNE

2. Clustering – basic concepts

Clustering (German: cluster analysis) refers to the grouping of data points in such a way that points within a group (clusters) are as similar as possible and points from different groups are as dissimilar as possible.

Distance measurements

Similarity is usually defined by a distance:

Euclidean distance (most common choice):

d(A, B) = √( Σ (aᵢ - bᵢ)² )

Manhattan distance (more robust for outliers):

d(A, B) = Σ |aᵢ - bᵢ|

Cosine similarity (for texts and vectors):

cos(A, B) = (A · B) / (‖A‖ · ‖B‖)

⚠️ Important: Before clustering, features must be scaled (e.g. with StandardScaler), otherwise features with a large range of values ​​will dominate the distance calculation.

Evaluate the quality of clustering

Since there are no labels, we need other evaluation metrics:

3. Algorithm: K-Means

3.1 Basic idea

K-Means divides the data into k clusters by iteratively adjusting cluster centers (centroids).

3.2 Algorithm step by step

1. Choose k random points as initial centroids

2. Repeat until convergence:
a) Assignment: Assign each data point to the nearest centroid
b) Update: Calculate new centroids as the average of the assigned points

3. Stop when the Centroids stop changing

Visualization of an iteration step:

Iteration 1: Iteration 2: Iteration 3 (converges):
× · · × · · · · ·
· · × · · · × ·· ···×··
· · × · · × · ·×

× = Centroid = Data point (color = assigned cluster)

3.3 Implementation

import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score

# Create synthetic data (3 clusters)
X, y_true = make_blobs(n_samples=300, centers=3, cluster_std=0.8, random_state=42)

# Scale
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# K-Means model
kmeans = KMeans(n_clusters=3, n_init=10, random_state=42)
kmeans.fit(X_scaled)

labels     = kmeans.labels_
centroids  = kmeans.cluster_centers_
inertia    = kmeans.inertia_

print(f"Inertia:          {inertia:.2f}")
print(f"Silhouette Score: {silhouette_score(X_scaled, labels):.3f}")

# Visualization
plt.figure(figsize=(8, 5))
plt.scatter(X_scaled[:, 0], X_scaled[:, 1], c=labels, cmap='tab10', alpha=0.6, s=40)
plt.scatter(centroids[:, 0], centroids[:, 1], c='black', marker='X', s=200, label='Centroids')
plt.title('K-Means Clustering (k=3)')
plt.legend()
plt.tight_layout()
plt.show()

3.4 Strengths and Weaknesses

4. Algorithm: DBSCAN

4.1 Basic idea

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) finds clusters based on local data density. It can detect arbitrarily shaped clusters and automatically marks points in sparsely populated regions as outliers (noise).

4.2 Key terms

• ε (eps): Maximum radius for the neighborhood of a point
• minPts: Minimum number of points in the ε neighborhood for a point to be a Core Point
• Core Point: Has at least minPts neighbors in radius ε
• Border Point: Located within the radius of a core point, but has too few neighbors
• Noise Point: Neither Core nor Border → Outlier, Label = -1

4.3 Implementation

from sklearn.cluster import DBSCAN
from sklearn.datasets import make_moons
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
import matplotlib.pyplot as plt

# Crescent data (K-Means would fail here)
X, _ = make_moons(n_samples=300, noise=0.08, random_state=42)
X_scaled = StandardScaler().fit_transform(X)

# DBSCAN
db = DBSCAN(eps=0.3, min_samples=5)
labels = db.fit_predict(X_scaled)

n_cluster  = len(set(labels)) - (1 if -1 in labels else 0)
n_noise    = (labels == -1).sum()

print(f"Gefundene Cluster: {n_cluster}")
print(f"Outliers:         {n_noise}")
if n_cluster > 1:
    mask = labels != -1
    print(f"Silhouette Score:  {silhouette_score(X_scaled[mask], labels[mask]):.3f}")

# Visualization
plt.scatter(X_scaled[:, 0], X_scaled[:, 1],
            c=labels, cmap='tab10', alpha=0.7, s=40)
# Mark outliers separately
noise_mask = labels == -1
plt.scatter(X_scaled[noise_mask, 0], X_scaled[noise_mask, 1],
            c='red', marker='x', s=80, label='Outliers')
plt.title(f'DBSCAN: {n_cluster} Cluster, {n_noise} Outliers')
plt.legend()
plt.tight_layout()
plt.show()

4.4 Strengths and Weaknesses

5. Algorithm: Hierarchical Clustering

5.1 Basic idea

Hierarchical clustering builds a tree structure (dendrogram) that shows all mergers of clusters. There is no need to specify k in advance - the number of clusters is determined by a cut in the dendrogram.

Agglomerative process (bottom-up):

1. Each data point = own cluster
2. Merge the two most similar clusters
3. Repeat until only 1 cluster remains

5.2 Linkage methods

How is the distance between two clusters calculated?

5.3 Implementation with Dendrogram

from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
import numpy as np

X, _ = make_blobs(n_samples=50, centers=3, random_state=42)

# Draw dendrogram
linked = linkage(X, method='ward')
plt.figure(figsize=(12, 5))
dendrogram(linked, leaf_rotation=90, leaf_font_size=8)
plt.axhline(y=10, color='red', linestyle='--', label='Schnitt → 3 Cluster')
plt.title('Dendrogram – Hierarchisches Clustering')
plt.xlabel('Datenpunkt-Index')
plt.ylabel('Distanz')
plt.legend()
plt.tight_layout()
plt.show()

# Extract cluster labels (cut at k=3)
hc = AgglomerativeClustering(n_clusters=3, linkage='ward')
labels = hc.fit_predict(X)
print(f"Cluster sizes: {np.bincount(labels)}")

6. Determine the number of clusters

6.1 Elbow method

The inertia is calculated for different k values. At the "elbow" point the curve flattens out - this is a good indication of the optimal number.

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt

X, _ = make_blobs(n_samples=300, centers=4, random_state=42)

inertias = []
k_range = range(1, 11)

for k in k_range:
    km = KMeans(n_clusters=k, n_init=10, random_state=42)
    km.fit(X)
    inertias.append(km.inertia_)

plt.plot(k_range, inertias, marker='o', color='#F6A01D')
plt.axvline(x=4, color='red', linestyle='--', label='Optimales k=4')
plt.xlabel('Anzahl Cluster k')
plt.ylabel('Inertia (SSE)')
plt.title('Elbow method zur Bestimmung von k')
plt.legend()
plt.tight_layout()
plt.show()

6.2 Silhouette analysis

The Silhouette Score measures how well each point fits its cluster:

s(i) = (b(i) - a(i)) / max(a(i), b(i))

a(i) = mean distance to points in your own cluster
b(i) = mean distance to points in the next cluster

• s(i) ≈ +1: Point is well in the cluster
• s(i) ≈ 0: Point lies on the border of two clusters
• s(i) ≈ -1: Point is probably in the wrong cluster

from sklearn.metrics import silhouette_score, silhouette_samples
import numpy as np

scores = []
for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=42)
    labels = km.fit_predict(X)
    scores.append(silhouette_score(X, labels))
    print(f"k={k}: Silhouette Score = {scores[-1]:.3f}")

best_k = range(2, 8)[np.argmax(scores)]
print(f"\nBestes k: {best_k}")

7. Dimensional Reduction – Basic Concepts

Real-world datasets often have hundreds or thousands of features (high dimensionality). This causes several problems:

The Curse of Dimensionality

• Lack of data: In high dimensional space the data set becomes very sparse
• Distance measurements fail: All points are placed at a similar distance from each other
• Overfitting: Models learn noise instead of real patterns
• Visualization: Can no longer be displayed directly from 4 dimensions onwards

Goals of dimension reduction

Many features (e.g. 100)
│
▼
Dimensional reduction
│
▼
Few features (e.g. 2-3)

We want to retain as much information as possible!

Applications:

• Visualize data (2D/3D)
• Remove noise (denoising)
• Reduce training time
• Eliminate multicollinearity (combine correlated features)

8. Algorithm: PCA – Principal Component Analysis

8.1 Basic idea

PCA looks for the directions in the data set that have the largest variance - these directions are called Principal Components. The data is then projected onto these directions.

Intuition: Imagine a cloud of points in 3D space that actually lies on a plane. PCA finds this plane and projects all the points into it - 3D becomes 2D without losing much information.

8.2 Mathematical background

1. Centering: Subtract mean from X
2. Calculate covariance matrix: C = (1/n) · Xᵀ · X
3. Eigenvalue decomposition: Eigenvectors = principal components, eigenvalues = explained variance
4. Projection: Project data onto the k most important principal components

8.3 Explained variance

A key concept: What percentage of the original information is retained?

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler

# Digit dataset: 1797 images, 64 features (8×8 pixels)
digits = load_digits()
X = StandardScaler().fit_transform(digits.data)

# PCA – calculate all components
pca_full = PCA()
pca_full.fit(X)

# Explained variance per component and cumulative
explained = pca_full.explained_variance_ratio_
cumulative = np.cumsum(explained)

plt.figure(figsize=(9, 4))
plt.subplot(1, 2, 1)
plt.bar(range(1, 21), explained[:20], color='#F6A01D')
plt.xlabel('Component')
plt.ylabel('Explained variance')
plt.title('Varianz pro Component')

plt.subplot(1, 2, 2)
plt.plot(range(1, len(cumulative)+1), cumulative, color='#F6A01D', marker='.')
plt.axhline(y=0.95, color='red', linestyle='--', label='95% variance')
plt.xlabel('Anzahl Componentn')
plt.ylabel('Cumulative explained variance')
plt.title('Cumulative variance')
plt.legend()
plt.tight_layout()
plt.show()

# How many components for 95%?
n_95 = np.argmax(cumulative >= 0.95) + 1
print(f"Components for 95% variance: {n_95} (instead of 64)")

8.4 PCA for visualization

# Reduction to 2 components for visualization
pca_2d = PCA(n_components=2)
X_2d = pca_2d.fit_transform(X)

plt.figure(figsize=(8, 6))
scatter = plt.scatter(X_2d[:, 0], X_2d[:, 1],
                      c=digits.target, cmap='tab10', alpha=0.6, s=20)
plt.colorbar(scatter, label='Digit')
plt.xlabel(f'PC1 ({pca_2d.explained_variance_ratio_[0]:.1%} Varianz)')
plt.ylabel(f'PC2 ({pca_2d.explained_variance_ratio_[1]:.1%} Varianz)')
plt.title('Zifferndatensatz – PCA auf 2 Componentn')
plt.tight_layout()
plt.show()

8.5 PCA as preprocessing

PCA can be used before a classification algorithm to reduce dimensionality:

from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.data, digits.target

# Pipeline: Scaling → PCA → kNN
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('pca',    PCA(n_components=30)),   # 64 → 30 Features
    ('knn',    KNeighborsClassifier(n_neighbors=5))
])

scores = cross_val_score(pipeline, X, y, cv=5)
print(f"CV Accuracy mit PCA (30 Komp.): {scores.mean():.2%}")

# Comparison without PCA
pipeline_nopca = Pipeline([
    ('scaler', StandardScaler()),
    ('knn',    KNeighborsClassifier(n_neighbors=5))
])
scores_nopca = cross_val_score(pipeline_nopca, X, y, cv=5)
print(f"CV Accuracy ohne PCA (64 Feat): {scores_nopca.mean():.2%}")

8.6 Strengths and Weaknesses

9. Algorithm: t-SNE

9.1 Basic idea

t-SNE (t-distributed stochastic neighbor embedding) is a nonlinear dimensionality reduction method specifically designed for visualizing high-dimensional data. Unlike PCA, t-SNE attempts to preserve the local structure (neighborhoods).

Attention: t-SNE is only suitable for visualization, not as preprocessing for downstream algorithms!

9.2 Implementation

from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

digits = load_digits()
X = StandardScaler().fit_transform(digits.data)

# t-SNE (takes a little longer than PCA)
tsne = TSNE(n_components=2, perplexity=30, n_iter=1000, random_state=42)
X_tsne = tsne.fit_transform(X)

plt.figure(figsize=(9, 7))
scatter = plt.scatter(X_tsne[:, 0], X_tsne[:, 1],
                      c=digits.target, cmap='tab10', alpha=0.7, s=20)
plt.colorbar(scatter, label='Digit')
plt.title('Digit dataset – t-SNE visualization')
plt.xlabel('t-SNE Dimension 1')
plt.ylabel('t-SNE Dimension 2')
plt.tight_layout()
plt.show()

9.3 PCA vs. t-SNE

Tip: For large data sets, first reduce to 50 components using PCA, then use t-SNE - this significantly saves computing time.

# Recommended combination for large data sets:
pca_pre = PCA(n_components=50)
X_pca   = pca_pre.fit_transform(X)
X_tsne  = TSNE(n_components=2, random_state=42).fit_transform(X_pca)

10. Application example: customer segmentation

10.1 Task

An online shop wants to divide its customers into segments based on their purchasing behavior in order to carry out targeted marketing measures.

10.2 Full pipeline

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.metrics import silhouette_score

# ── 1. Create synthetic customer data ─────────────────────────────────
np.random.seed(42)
n = 400

# Simulate three customer groups
groups = {
    'Occasional buyers':   {'n': 150, 'spending': (20, 10),  'visits': (2, 1)},
    'Regular customers':           {'n': 150, 'spending': (80, 20),  'visits': (8, 2)},
    'Premium customers':        {'n': 100, 'spending': (200, 40), 'visits': (5, 1)},
}

dfs = []
for name, params in groups.items():
    n_g = params['n']
    df_g = pd.DataFrame({
        'monthly_spending':  np.random.normal(*params['spending'], n_g).clip(5),
        'visits_pro_monat':    np.random.normal(*params['visits'],  n_g).clip(1),
        'customer_satisfaction':  np.random.uniform(2, 5, n_g).round(1),
        'days_since_last_purchase':  np.random.exponential(15, n_g).clip(1, 90).astype(int),
    })
    dfs.append(df_g)

customers = pd.concat(dfs, ignore_index=True).sample(frac=1, random_state=42)
print(customers.describe().round(2))

# ── 2. Preprocessing ─────────────────────────── ───────────────────────────
X = customers.values
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# ── 3. Determine optimal k (elbow + silhouette) ─────────────────────────
inertias, silhouettes = [], []
k_range = range(2, 9)

for k in k_range:
    km = KMeans(n_clusters=k, n_init=10, random_state=42)
    labels = km.fit_predict(X_scaled)
    inertias.append(km.inertia_)
    silhouettes.append(silhouette_score(X_scaled, labels))

fig, axes = plt.subplots(1, 2, figsize=(11, 4))
axes[0].plot(k_range, inertias, marker='o', color='#F6A01D')
axes[0].set_title('Elbow method'); axes[0].set_xlabel('k'); axes[0].set_ylabel('Inertia')
axes[1].plot(k_range, silhouettes, marker='o', color='#F6A01D')
axes[1].set_title('Silhouette Score'); axes[1].set_xlabel('k'); axes[1].set_ylabel('Score')
plt.tight_layout(); plt.show()

# ── 4. K-Means with k=3 ────────────────────────── ──────────────────────────
kmeans = KMeans(n_clusters=3, n_init=10, random_state=42)
customers['segment'] = kmeans.fit_predict(X_scaled)

# ── 5. Analyze segments ─────────────────────── ────────────────────────
print("\nSegment profiles:")
print(customers.groupby('segment').mean().round(2))

# ── 6. Visualization with PCA ────────────────────── ───────────────────────
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_scaled)

plt.figure(figsize=(8, 6))
for seg in sorted(customers['segment'].unique()):
    mask = customers['segment'] == seg
    plt.scatter(X_2d[mask, 0], X_2d[mask, 1], label=f'Segment {seg}', alpha=0.6, s=40)

# Drawing centroids in PCA space
centroids_2d = pca.transform(kmeans.cluster_centers_)
plt.scatter(centroids_2d[:, 0], centroids_2d[:, 1],
            c='black', marker='X', s=200, zorder=5, label='Centroids')
plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.1%})')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.1%})')
plt.title('Customer segmentation – K-Means (k=3) visualisiert mit PCA')
plt.legend(); plt.tight_layout(); plt.show()

11. Comparison of algorithms

Clustering algorithms

Dimensional reduction methods

12. Summary and outlook

Overview

Unsupervised learning
│
├── Clustering
│ ├── K-Means → fast, compact clusters, k required in advance
│ ├── DBSCAN → any shapes, outliers, no k necessary
│ └── Hierarchical → Dendrogram, exploratory, slow
│
└── Dimensional reduction
├── PCA → linear, fast, preprocessing + visualization
└── t-SNE → nonlinear, slow, visualization only

Decision support

Clustering or dimensionality reduction?
→ Group data: clustering
→ Visualize/simplify data: Dimension reduction

Which clustering algorithm?
→ I know how many clusters: K-Means
→ Unknown cluster form: DBSCAN
→ Exploratory analysis: Hierarchical

Which dimension reduction?
→ As preprocessing: PCA
→ Visualization only: t-SNE (possibly after PCA)

Checklist

✅ Always scale features before clustering/PCA (StandardScaler)
✅ Determine the number of clusters with Elbow + Silhouette
✅ Always visualize the result (PCA or t-SNE)
✅ Interpret cluster content (mean values ​​per cluster)
✅ PCA: check explained variance (target: 90-95%)
✅ t-SNE only for visualization, not as preprocessing

Outlook: Next topics

• Deep Learning: Neural networks, backpropagation, activation functions
• Convolutional Neural Networks (CNNs): Image recognition
• Natural Language Processing: Text classification, transformer models
• MLOps: Version, deploy, and monitor models

Appendix: Quick Reference

# ── Clustering ────────────────────────────── ──────────────────────────────
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering

km = KMeans(n_clusters=3, n_init=10, random_state=42)
db = DBSCAN(eps=0.5, min_samples=5)
hc = AgglomerativeClustering(n_clusters=3, linkage='ward')

labels = km.fit_predict(X_scaled)   # Same for all three

# ── Rating ────────────────────────────── ───────────────────────────────
from sklearn.metrics import silhouette_score, davies_bouldin_score

sil = silhouette_score(X_scaled, labels)   # higher = better
db_score = davies_bouldin_score(X_scaled, labels)  # lower = better

# ── Dimensional reduction ───────────────────────── ──────────────────────────
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE

pca   = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print(pca.explained_variance_ratio_)   # Varianzanteil je Component

tsne   = TSNE(n_components=2, perplexity=30, random_state=42)
X_tsne = tsne.fit_transform(X_scaled)  # Nur fit_transform, kein transform!

# ── Dendrogram ────────────────────────────── ──────────────────────────────
from scipy.cluster.hierarchy import dendrogram, linkage
linked = linkage(X_scaled, method='ward')
dendrogram(linked)

HTL Pinkafeld – IF/IT | Data Science & AI | 12th grade