Machine Learning – Basics and Supervised Learning

Subject: Data Science & Artificial Intelligence
School level: 12th grade – HTL Informatik
Requirements: Python, C, JavaScript, HTML/CSS
Author: HTL Pinkafeld – IF/IT

1. What is Machine Learning?

Machine learning (ML) is a branch of artificial intelligence (AI). Instead of programming explicit rules into a computer, it learns from data to recognize patterns and make decisions.

Classical programming vs. machine learning

Application examples in everyday life

• Spam filter: Emails are automatically recognized as spam
• Image recognition: Face recognition on the smartphone
• Recommendation systems: Netflix suggests shows you might like
• Voice Assistants: Siri, Alexa, Google Assistant understand natural language
• Medicine: Detection of tumors on x-ray images

💡 Key idea: Machine learning is not magic – it is statistics and optimization on large amounts of data.

2. Types of Machine Learning

There are three basic learning paradigms:

2.1 Supervised Learning

The model learns from labeled data - each training example has a known answer (label).

Input (feature): living space, number of rooms, location
Output (label): Price of the apartment

→ The model learns the mapping f(X) → y

Typical tasks:

• Classification: Assign input to a category (e.g. spam / not spam)
• Regression: Predict a continuous value (e.g. house price)

2.2 Unsupervised Learning

There are no labels. The model independently searches for structures and patterns in the data.

Typical tasks:

• Clustering: Group similar data points (e.g. customer segmentation)
• Dimension reduction: Simplify data (e.g. PCA)

2.3 Reinforcement Learning

An agent learns through trial and error in an environment. He receives rewards for good actions.

Examples: Chess AI (AlphaZero), autonomous driving, robot control

3. The ML workflow

Every ML project follows a typical flow:

1. Define the problem
↓
2. Collect & understand data
↓
3. Process data (preprocessing)
↓
4. Select model & train
↓
5. Evaluate model
↓
6. Optimize the model (hyperparameter tuning)
↓
7. Deploy & monitor model

Important libraries in Python

4. Data – the heart of ML

4.1 Terms

• Feature: An input variable (e.g. age, size, color)
• Label (target size): The output value you are looking for
• Dataset: Set of all examples
• Feature Matrix X: Table with all input features (rows = examples, columns = features)
• Label vector y: Column with the target values

4.2 Splitting data

To help us assess how well our model responds to unknown data, we split the dataset:

In scikit-learn:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

⚠️ Important: Test data must never be seen by the model during training!

4.3 Preprocessing – prepare data

Raw data is rarely perfect. Typical problems:

• Missing values: Remove or fill lines with NaN
• Categorical variables: Convert text to numbers (one-hot encoding)
• Scaling: Bring features to the same size (normalization)

import pandas as pd
from sklearn.preprocessing import StandardScaler

df = pd.read_csv('daten.csv')

# Fill in missing values ​​with average
df['alter'].fillna(df['alter'].mean(), inplace=True)

# Normalization (mean 0, standard deviation 1)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

5. Supervised Learning – Basic Concepts

5.1 Classification vs. Regression

Classification: "Which category?"
Input → discrete class (0/1, spam/no spam, tier A/B/C)

Regression: "What value?"
Input → continuous value (€, °C, kg)

5.2 Learning process

In supervised learning, the model minimizes a loss function that measures how large the error is between the prediction and the actual label.

• Regression: Mean Squared Error (MSE)
• Classification: Cross-Entropy, Accuracy

6. Algorithm: k-Nearest Neighbors (kNN)

6.1 Basic idea

"Tell me who your neighbors are and I'll tell you who you are."

kNN classifies a new data point based on the k most similar training examples. The class of the majority of k neighbors is used as the prediction.

6.2 Distance measurements

“Similarity” is measured by a distance. Most common: Euclidean distance:

d(A, B) = √( (x₁-x₂)² + (y₁-y₂)² )

6.3 Choice of k

Rule of thumb: k = √n (where n = number of training examples), avoid even numbers in binary classification.

6.4 Practical example: Iris data set

The iris dataset contains 150 flowers from 3 iris species, each with 4 features (sepal/corolla length and width).

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

# 1. Load data
iris = load_iris()
X, y = iris.data, iris.target

# 2. Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 3. Create and train the model
model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)

# 4. Predictions
y_pred = model.predict(X_test)

# 5. Rate
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2%}")

6.5 Strengths and Weaknesses

7. Algorithm: Decision Tree

7.1 Basic idea

A decision tree recursively splits the data based on conditions (splits) until unique classes emerge. Every inner node is a question, every leaf a prediction.

Example: Will I play tennis?

Weather?
/\
sunny rain
|           |
Humidity?   NO
/\
<70% ≥70%
YES NO

7.2 Splitting criteria

How does the algorithm choose the best split?

• Gini-Impurity: Measure of impurity of a node (0 = pure, 0.5 = maximum mixed)
• Information Gain (Entropy): Based on the concept of entropy from information theory

Gini(t) = 1 - Σ p(i|t)²

7.3 Practical example

from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

# Create model (max_depth limits the depth → prevents overfitting)
dt = DecisionTreeClassifier(max_depth=3, random_state=42)
dt.fit(X_train, y_train)

# Output tree as text
print(export_text(dt, feature_names=iris.feature_names))

print(f"Accuracy: {dt.score(X_test, y_test):.2%}")

7.4 Feature Importance

A big advantage of Decision Trees: Interpretability. You can see which features are important.

import pandas as pd

importance = pd.Series(
    dt.feature_importances_,
    index=iris.feature_names
).sort_values(ascending=False)

print(importance)

7.5 Strengths and Weaknesses

💡 Note: In practice, Random Forests are often used - combining many trees that together deliver better results.

8. Algorithm: Linear Regression

8.1 Basic idea

Linear regression models the relationship between one or more features and a continuous target variable using a straight line (or hyperplane).

Simple linear regression (1 feature):

ŷ = w₀ + w₁ x

Multiple linear regression (multiple features):

ŷ = w₀ + w₁ x₁ + w₂ x₂ + ... + wₙ xₙ

The parameters w (weights/coefficients) are learned in such a way that the Mean Square Error (MSE) is minimized:

MSE = (1/n) · Σ (yᵢ - ŷᵢ)²

8.2 Practical example: Housing prices

import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

# Synthetic data: living space (m²) → price (€1000)
np.random.seed(42)
flaeche = np.random.uniform(40, 150, 100).reshape(-1, 1)
preis = 1.5 * flaeche.flatten() + np.random.normal(0, 15, 100) + 50

# Train model
model = LinearRegression()
model.fit(flaeche, preis)

# Coefficients
print(f"Steigung (w₁): {model.coef_[0]:.2f}")
print(f"y-Achsenabschnitt (w₀): {model.intercept_:.2f}")
print(f"R²-Score: {r2_score(preis, model.predict(flaeche)):.3f}")

# Visualization
plt.scatter(flaeche, preis, alpha=0.5, label='Datenpunkte')
plt.plot(flaeche, model.predict(flaeche), color='red', label='Regressionsgerade')
plt.xlabel('Living space (m²)')
plt.ylabel('Preis (€1000)')
plt.legend()
plt.title('Lineare Regression: Wohnungspreise')
plt.show()

8.3 The R² score

The R² score (coefficient of determination) indicates how much variance in the data is explained by the model:

• R² = 1.0 → perfect model
• R² = 0.0 → Model is as good as the mean
• R² < 0 → Model worse than the mean

8.4 Strengths and Weaknesses

9. Algorithm: Logistic Regression

9.1 Basic idea

Despite the name, logistic regression is a classification algorithm. It predicts the probability that a data point belongs to a class.

The output of linear regression (any value) is mapped to the range [0, 1] by the sigmoid function:

σ(z) = 1 / (1 + e⁻ᶻ)

From a threshold value (usually 0.5) the following are classified:

• P(y=1) ≥ 0.5 → Class 1
• P(y=1) < 0.5 → Class 0

9.2 Practical example: Titanic data set

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# Simplified Titanic dataset
data = {
    'alter': [22, 38, 26, 35, 28, 54, 2, 27, 14, 4],
    'klasse': [3, 1, 3, 1, 3, 1, 3, 3, 2, 3],
    'gender': [0, 1, 1, 1, 0, 0, 0, 1, 1, 1],  # 0=male, 1=female
    'ueberlebt': [0, 1, 1, 1, 0, 0, 0, 0, 1, 1]
}
df = pd.DataFrame(data)

X = df[['alter', 'klasse', 'gender']]
y = df['ueberlebt']

# On a small amount of data: no train/test split for this example
model = LogisticRegression()
model.fit(X, y)

# Prediction for a new person
neue_person = [[30, 2, 1]]  # 30 Jahre, 2. Klasse, weiblich
prob = model.predict_proba(neue_person)[0]
print(f"Survival probability: {prob[1]:.1%}")

9.3 Decision boundary

Logistic regression learns a linear decision boundary in feature space. Data points on one side are classified as Class 0 and on the other as Class 1.

9.4 Strengths and Weaknesses

10. Model evaluation and metrics

10.1 Classification metrics

Confusion Matrix

Key metrics

Accuracy = (TP + TN) / (TP + TN + FP + FN)
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
F1 score = 2 (Precision Recall) / (Precision + Recall)

When which metric?

• Accuracy: Balanced classes, general assessment
• Precision: If false positives are expensive (e.g. spam filters: do not delete legitimate emails)
• Recall: When false negatives are expensive (e.g. cancer diagnosis: no sick people missed)
• F1 Score: Balanced measure between precision and recall

from sklearn.metrics import classification_report, confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt

# According to the prediction...
print(classification_report(y_test, y_pred, target_names=iris.target_names))

# Visualize Confusion Matrix
cm = confusion_matrix(y_test, y_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
            xticklabels=iris.target_names,
            yticklabels=iris.target_names)
plt.title('Confusion Matrix')
plt.show()

10.2 Regression metrics

10.3 Cross Validation

To evaluate the model more robustly, k-fold cross-validation is used: The data is divided into k parts, the model is trained and evaluated k times.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
print(f"CV Accuracy: {scores.mean():.2%} ± {scores.std():.2%}")

11. Overfitting and underfitting

11.1 The bias-variance tradeoff

A central problem in machine learning:

Underfitting: Good model: Overfitting:
Too simple Balanced Too complex

Training: bad Training: good Training: perfect
Test: bad Test: good Test: bad

High bias balance high variance

11.2 Causes and Solutions

11.3 Regularization

Regularization penalizes complex models to reduce overfitting:

• L1 (Lasso): Sets unimportant coefficients to 0 → Feature Selection
• L2 (Ridge): Reduces all coefficients evenly

from sklearn.linear_model import Ridge, Lasso

ridge = Ridge(alpha=1.0)  # alpha = regularization strength
lasso = Lasso(alpha=0.1)

11.4 Learning curves

Learning curves show whether a model is underfitted or overfitted:

from sklearn.model_selection import learning_curve
import numpy as np
import matplotlib.pyplot as plt

train_sizes, train_scores, val_scores = learning_curve(
    model, X, y, cv=5, train_sizes=np.linspace(0.1, 1.0, 10)
)

plt.plot(train_sizes, train_scores.mean(axis=1), label='Training')
plt.plot(train_sizes, val_scores.mean(axis=1), label='Validierung')
plt.xlabel('Training size')
plt.ylabel('Accuracy')
plt.legend()
plt.title('Lernkurven')
plt.show()

12. Summary and outlook

Overview of the algorithms covered

Choosing the right algorithm

Is the target feature continuous?
→ YES: Regression (Linear Regression, ...)
→ NO: Classification

How much data is there?
→ Little data: kNN, logistic regression
→ Lots of data: decision trees, random forests, neural networks

Does the model have to be interpretable?
→ YES: Decision Tree, Linear/Logistic Regression
→ NO: All algorithms possible

Outlook: Next topics

• Ensemble methods: Random Forest, Gradient Boosting (XGBoost)
• Unsupervised Learning: K-Means Clustering, PCA
• Deep Learning: Neural Networks, CNNs, RNNs
• Natural Language Processing: Text classification, transformer models
• MLOps: Bring and monitor models into production

Appendix: Quick reference scikit-learn

# Uniform API pattern for all scikit-learn models:
from sklearn.XXX import YYYModel

model = YYYModel(hyperparameter1=wert1)   # 1. Erstellen
model.fit(X_train, y_train)               # 2. Trainieren
y_pred = model.predict(X_test)            # 3. Vorhersagen
score = model.score(X_test, y_test)       # 4. Bewerten

Important imports

# Data
from sklearn.datasets import load_iris, load_diabetes, make_classification
from sklearn.model_selection import train_test_split, cross_val_score

# Preprocessing
from sklearn.preprocessing import StandardScaler, MinMaxScaler, LabelEncoder
from sklearn.impute import SimpleImputer

# Algorithms
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge, Lasso

# Metrics
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report,
    mean_squared_error, mean_absolute_error, r2_score
)

HTL Pinkafeld – IF/IT | Data Science & AI | 12th grade