Deep Learning – Neural Networks and CNNs

Subject: Data Science & Artificial Intelligence
School level: 12th grade – HTL Informatik
Prerequisites: Supervised Learning, Python / NumPy, basic linear algebra
Author: HTL Pinkafeld – IF/IT

1. From ML to Deep Learning

1.1 What is Deep Learning?

Deep learning is a subfield of machine learning that is based on artificial neural networks with many layers. The term "deep" refers to the number of layers - modern networks often have dozens to hundreds of them.

1.2 Why Deep Learning?

Classic ML algorithms like Random Forest or SVM require handcrafted features (feature engineering). With images, audio or text, this is extremely time-consuming - you would have to manually tell the algorithm what to pay attention to.

Deep learning learns the relevant features automatically from the raw data:

Classic ML:
Raw data → [Feature engineering, manual] → Features → Algorithm → Output

Deep learning:
Raw data ────────────────────────────────→ Neural network → Output
(learns features himself)

1.3 When does deep learning make sense?

1.4 Historic breakthrough

1958 Perceptron (Rosenblatt)
1986 Backpropagation (Rumelhart, Hinton)
1998 LeNet – first CNN for digit recognition (LeCun)
2012 AlexNet wins ImageNet – Deep Learning Revolution
2017 Transformer Architecture (Vaswani et al.)
2022 ChatGPT – Large Language Models in the mainstream

2. The artificial neuron

2.1 Biological model

The artificial neuron is inspired by the biological neuron:

Biological: Artificial:
Dendrites → inputs x₁, x₂, ..., xₙ
Cell body → Weighted sum + bias
Axon → activation function → output

2.2 Mathematical model

A single neuron calculates:

z = w₁ x₁ + w₂ x₂ + ... + wₙ xₙ + b
= wᵀ x + b (vector notation)

y = f(z) (apply activation function)

• x = input vector (features)
• w = weight vector (learned during training)
• b = Bias (shift, is also learned)
• f = activation function (brings nonlinearity to the network)

2.3 The Perceptron

The simplest neuron - the Perceptron - uses a step function as activation:

import numpy as np

def perceptron(x, w, b):
    z = np.dot(w, x) + b
    return 1 if z >= 0 else 0

# Example: AND gate
w = np.array([1.0, 1.0])
b = -1.5

print(perceptron([0, 0], w, b))   # 0
print(perceptron([0, 1], w, b))   # 0
print(perceptron([1, 0], w, b))   # 0
print(perceptron([1, 1], w, b))   # 1

⚠️ Problem: A single neuron can only solve linearly separable problems. The XOR problem, for example, cannot be separated linearly - we need several layers for that.

3. Activation functions

Activation functions bring nonlinearity to the network. Without them, a multi-level network would be mathematically equivalent to a single neuron.

3.1 Overview

3.2 ReLU – Rectified Linear Unit

ReLU is today the default activation function in hidden layers:

import numpy as np
import matplotlib.pyplot as plt

z = np.linspace(-4, 4, 200)

relu        = np.maximum(0, z)
leaky_relu  = np.where(z > 0, z, 0.01 * z)
sigmoid     = 1 / (1 + np.exp(-z))
tanh        = np.tanh(z)

fig, axes = plt.subplots(1, 2, figsize=(10, 4))

axes[0].plot(z, relu,       label='ReLU',       color='#F6A01D', linewidth=2)
axes[0].plot(z, leaky_relu, label='Leaky ReLU', color='#C97D00', linestyle='--', linewidth=2)
axes[0].axhline(0, color='gray', linewidth=0.5)
axes[0].axvline(0, color='gray', linewidth=0.5)
axes[0].set_title('ReLU Varianten'); axes[0].legend()

axes[1].plot(z, sigmoid, label='Sigmoid', color='#F6A01D', linewidth=2)
axes[1].plot(z, tanh,    label='Tanh',    color='#C97D00', linestyle='--', linewidth=2)
axes[1].axhline(0, color='gray', linewidth=0.5)
axes[1].set_title('Sigmoid & Tanh'); axes[1].legend()

plt.tight_layout()
plt.show()

3.3 Why ReLU is so popular

• Easy to calculate: Just a comparison with 0
• No Vanishing Gradient: Gradient is constant 1 for z > 0
• Sparse Activation: Many neurons are inactive (z < 0) → more efficient representation

Vanishing Gradient Problem: With Sigmoid/Tanh, the gradients in deep networks become smaller and smaller (towards 0) - the network hardly learns anymore. ReLU largely solves this problem.

4. Feedforward Neural Network

4.1 Structure

A feedforward neural network (also Multi-Layer Perceptron, MLP) consists of:

Input Hidden Layers Output
layer (hidden layers) layer

x₁ ──┐ ┌── ŷ₁
x₂ ──┼─→ [Layer 1] → [Layer 2] ─┼── ŷ₂
x₃ ──┘ └── ŷ₃

Data always flows forward (feedforward)

• Input layer: One neuron per feature
• Hidden layers: Learning internal representations
• Output layer: One neuron per class (classification) or 1 neuron (regression)

4.2 Matrix notation of a layer

For a layer with n inputs and m neurons:

Z = X · Wᵀ + b
A = f(Z)

X: (batch_size × n) input
W: (m × n) weight matrix
b: (m,) bias vector
A: (batch_size × m) Output after activation

4.3 Universal approximation theorem

A neural network with a hidden layer and enough neurons can approximate any continuous function with arbitrary accuracy.

However, in practice, deep networks (many layers, fewer neurons per layer) are more efficient than wide networks (one layer, many neurons).

5. Training: Backpropagation and Gradient Descent

5.1 The learning principle

During training, an error measure (loss) is calculated and the weights are adjusted so that the error decreases. The heart of the matter is gradient descent:

w_new = w_old - η · ∂L/∂w

η = learning rate (how big is the step?)
∂L/∂w = gradient of loss relative to weight

5.2 Backpropagation

Backpropagation efficiently calculates the gradients of all weights using the chain rule of differential calculus - from the output layer backwards to the input layer.

Forward Pass:
x → Layer 1 → Layer 2 → ... → ŷ → L(ŷ, y)

Backward Pass:
∂L/∂wₙ ← ∂L/∂wₙ₋₁ ← ... ← ∂L/∂w₁
(apply chain rule backwards)

Update weights:
w ← w - η · ∂L/∂w

5.3 Gradient Descent Variants

# Mini-Batch: Typical batch sizes: 32, 64, 128, 256
model.fit(X_train, y_train, batch_size=32, epochs=50)

5.4 The learning rate

The learning rate η is one of the most important hyperparameters:

η too large: Skips minimum → does not converge
η too small: Learns very slowly → takes many epochs
η optimal: Converges reliably to the minimum

Tip: Learning rate schedulers adjust η during training - large at the beginning, small later.

6. Loss functions

The Loss Function measures how wrong the predictions are.

6.1 Overview

6.2 Cross-Entropy clearly

Prediction for cat: 90% → Loss small (almost right)
Prediction for cat: 10% → Loss big (very wrong)

Binary Cross Entropy:
L = -[y · log(ŷ) + (1-y) · log(1-ŷ)]

7. First network with Keras

7.1 Keras and TensorFlow

Keras is the official high-level API of TensorFlow. It allows you to quickly build and train neural networks with just a few lines of code.

pip install tensorflow

7.2 Classification: MNIST digit recognition

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# ── 1. Load data ──────────────────────────── ────────────────────────────
(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()

# Normalize: Pixel values ​​0-255 → 0.0-1.0
X_train = X_train.astype('float32') / 255.0
X_test  = X_test.astype('float32')  / 255.0

# Smooth images: (28, 28) → (784,)
X_train = X_train.reshape(-1, 784)
X_test  = X_test.reshape(-1, 784)

print(f"Trainingsdaten: {X_train.shape}")   # (60000, 784)
print(f"Testdaten:      {X_test.shape}")    # (10000, 784)

# ── 2. Build the model ────────────────────────── ──────────────────────────
model = keras.Sequential([
    layers.Dense(128, activation='relu',    input_shape=(784,)),
    layers.Dense(64,  activation='relu'),
    layers.Dense(10,  activation='softmax')   # 10 Klassen (0–9)
])

model.summary()

# ── 3. Compile ──────────────────────────── ────────────────────────────
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# ── 4. Train ──────────────────────────── ─────────────────────────────
history = model.fit(
    X_train, y_train,
    epochs=10,
    batch_size=64,
    validation_split=0.1,   # 10% der Trainingsdaten als Validierung
    verbose=1
)

# ── 5. Evaluate ──────────────────────────── ─────────────────────────────
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"\nTest Accuracy: {test_acc:.2%}")

# ── 6. Visualize training progress ─────────────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(11, 4))

axes[0].plot(history.history['loss'],     label='Training Loss')
axes[0].plot(history.history['val_loss'], label='Validation Loss')
axes[0].set_title('Verlust'); axes[0].legend()

axes[1].plot(history.history['accuracy'],     label='Training Accuracy')
axes[1].plot(history.history['val_accuracy'], label='Validation Accuracy')
axes[1].set_title('Accuracy'); axes[1].legend()

plt.tight_layout()
plt.show()

7.3 Read model architecture

Layer (type) Output Shape Param #
================================================
dense (Dense) (None, 128) 100,480 ← 784×128 + 128 Bias
dense_1 (Dense) (None, 64) 8,256 ← 128×64 + 64 Bias
dense_2 (Dense) (None, 10) 650 ← 64×10 + 10 Bias
================================================
Total params: 109,386

💡 Calculate number of parameters: Ausgaben × Eingaben + Ausgaben (Bias)

8. Combat overfitting

8.1 Dropout

Dropout randomly deactivates a portion of neurons during training. This prevents individual neurons from becoming too dominant.

model = keras.Sequential([
    layers.Dense(256, activation='relu', input_shape=(784,)),
    layers.Dropout(0.3),                  # 30% der Neuronen deaktivieren
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

During inference (prediction) all neurons are activated again - dropout is only active during training.

8.2 Batch Normalization

Batch Normalization normalizes the activations of each layer, which stabilizes and speeds up training:

model = keras.Sequential([
    layers.Dense(256, input_shape=(784,)),
    layers.BatchNormalization(),
    layers.Activation('relu'),
    layers.Dense(128),
    layers.BatchNormalization(),
    layers.Activation('relu'),
    layers.Dense(10, activation='softmax')
])

8.3 Early Stopping

Training stops automatically when the Validation Loss no longer decreases:

from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau

callbacks = [
    EarlyStopping(
        monitor='val_loss',
        patience=5,           # Stoppt nach 5 Epochen ohne Verbesserung
        restore_best_weights=True
    ),
    ReduceLROnPlateau(
        monitor='val_loss',
        factor=0.5,           # Lernrate halbieren
        patience=3
    )
]

history = model.fit(
    X_train, y_train,
    epochs=100,
    batch_size=64,
    validation_split=0.1,
    callbacks=callbacks,
    verbose=1
)

8.4 Regularization (L2)

from tensorflow.keras import regularizers

model = keras.Sequential([
    layers.Dense(256, activation='relu',
                 kernel_regularizer=regularizers.l2(0.001),
                 input_shape=(784,)),
    layers.Dense(10, activation='softmax')
])

8.5 Overview of regularization techniques

9. Convolutional Neural Networks (CNNs)

9.1 Motivation

An MLP on a 224x224 color image would already have 224 × 224 × 3 = 150,528 inputs in the first layer - that's millions of parameters before the network is even one layer deep. Furthermore, an MLP completely ignores the spatial structure of the image.

CNNs solve both problems through local connections and shared weights.

9.2 The convolution operator (convolution)

A filter (kernel) of size k×k slides over the image and calculates a weighted sum at each position:

Input (5×5): Filter (3×3): Output (Feature Map):
1 0 1 0 0 1 0 -1 · · ·
0 1 0 1 0 * 0 0 0 = 4
1 0 1 0 1 -1 0 1 · · ·
0 1 0 0 1
0 1 1 0 1

• Each filter learns a specific feature (edge, texture, shape)
• Many filters in parallel → many feature maps
• Weights are learned during training (like Dense shifts)

9.3 Important CNN layers

Conv2D – Convolutional Layer:

layers.Conv2D(
    filters=32,         # Anzahl der Filter
    kernel_size=(3, 3), # Filter size
    activation='relu',
    padding='same'      # Output size = input size
)

MaxPooling2D – reduction layer:

layers.MaxPooling2D(pool_size=(2, 2))
# Reduces width and height by a factor of 2
# Selects the maximum value in each 2x2 range

Flatten – Transition to Dense:

layers.Flatten()
# Converts 2D feature map to 1D vector

9.4 Typical CNN architecture

Input image (28×28×1)
│
Conv2D (32 filters, 3×3) → ReLU → (28×28×32)
│
MaxPooling (2×2) → (14×14×32)
│
Conv2D (64 filters, 3×3) → ReLU → (14×14×64)
│
MaxPooling (2×2) → (7×7×64)
│
Flatten → (3136,)
│
Dense (128) → ReLU
│
Dropout (0.5)
│
Dense (10) → Softmax → (10,)

9.5 CNN for MNIST

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Load and prepare data
(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()

# For CNNs: (n, 28, 28, 1) – Add channel dimension
X_train = X_train.reshape(-1, 28, 28, 1).astype('float32') / 255.0
X_test  = X_test.reshape(-1, 28, 28, 1).astype('float32') / 255.0

# CNN model
model = keras.Sequential([
    # Block 1
    layers.Conv2D(32, (3, 3), activation='relu', padding='same',
                  input_shape=(28, 28, 1)),
    layers.MaxPooling2D((2, 2)),

    # Block 2
    layers.Conv2D(64, (3, 3), activation='relu', padding='same'),
    layers.MaxPooling2D((2, 2)),

    # Classification header
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model.summary()

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

history = model.fit(
    X_train, y_train,
    epochs=10,
    batch_size=64,
    validation_split=0.1
)

test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"CNN Test Accuracy: {test_acc:.2%}")   # Typisch: > 99%

10. Practical Project: Image Classification using CIFAR-10

10.1 The CIFAR-10 data set

CIFAR-10 contains 60,000 color images (32×32 pixels) in 10 classes: Flugzeug, Auto, Vogel, Katze, Reh, Hund, Frosch, Pferd, Schiff, LKW

10.2 Data Augmentation

Since 50,000 training images are relatively few for a CNN, we artificially expand the data set using data augmentation - creating slightly modified copies:

from tensorflow.keras.layers import RandomFlip, RandomRotation, RandomZoom

augmentation = keras.Sequential([
    RandomFlip('horizontal'),
    RandomRotation(0.1),
    RandomZoom(0.1),
])

10.3 Full pipeline

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import matplotlib.pyplot as plt
import numpy as np

# ── 1. Load data ──────────────────────────── ────────────────────────────
(X_train, y_train), (X_test, y_test) = keras.datasets.cifar10.load_data()

X_train = X_train.astype('float32') / 255.0
X_test  = X_test.astype('float32')  / 255.0

klassen = ['Flugzeug', 'Auto', 'Vogel', 'Katze', 'Reh',
           'Hund', 'Frosch', 'Pferd', 'Schiff', 'LKW']

# ── 2. Model with data augmentation ──────────────────────────────────────
model = keras.Sequential([
    # Data augmentation (only active during training)
    layers.RandomFlip('horizontal'),
    layers.RandomRotation(0.1),
    layers.RandomZoom(0.1),

    # Block 1
    layers.Conv2D(32, (3, 3), activation='relu', padding='same',
                  input_shape=(32, 32, 3)),
    layers.BatchNormalization(),
    layers.Conv2D(32, (3, 3), activation='relu', padding='same'),
    layers.MaxPooling2D((2, 2)),
    layers.Dropout(0.25),

    # Block 2
    layers.Conv2D(64, (3, 3), activation='relu', padding='same'),
    layers.BatchNormalization(),
    layers.Conv2D(64, (3, 3), activation='relu', padding='same'),
    layers.MaxPooling2D((2, 2)),
    layers.Dropout(0.25),

    # Block 3
    layers.Conv2D(128, (3, 3), activation='relu', padding='same'),
    layers.BatchNormalization(),
    layers.MaxPooling2D((2, 2)),
    layers.Dropout(0.25),

    # Classification header
    layers.Flatten(),
    layers.Dense(256, activation='relu'),
    layers.BatchNormalization(),
    layers.Dropout(0.5),
    layers.Dense(10, activation='softmax')
])

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.001),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# ── 3. Train ──────────────────────────── ─────────────────────────────
callbacks = [
    keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=10,
                                   restore_best_weights=True),
    keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=5)
]

history = model.fit(
    X_train, y_train,
    epochs=50,
    batch_size=64,
    validation_split=0.1,
    callbacks=callbacks
)

# ── 4. Evaluate ──────────────────────────── ─────────────────────────────
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"Test Accuracy: {test_acc:.2%}")

# ── 5. View individual predictions ──────────────────────────────────────
y_pred = np.argmax(model.predict(X_test[:16]), axis=1)

fig, axes = plt.subplots(2, 8, figsize=(14, 4))
for i, ax in enumerate(axes.flat):
    ax.imshow(X_test[i])
    farbe = 'green' if y_pred[i] == y_test[i][0] else 'red'
    ax.set_title(klassen[y_pred[i]], color=farbe, fontsize=8)
    ax.axis('off')
plt.suptitle('CIFAR-10 predictions (green=correct, red=wrong)')
plt.tight_layout()
plt.show()

11. Optimizers in comparison

11.1 SGD with momentum

Momentum “remembers” the direction of recent updates and accelerates training in consistent directions:

optimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)

11.2 Adam

Adam (Adaptive Moment Estimation) is the standard optimizer for deep learning. It automatically adjusts the learning rate for each parameter:

optimizer = keras.optimizers.Adam(learning_rate=0.001)   # Standard

11.3 Comparison

12. Transfer learning

12.1 Basic idea

Large pre-trained networks (e.g. VGG16, ResNet, MobileNet) have been trained on millions of images and learned to recognize common image features (edges, shapes, textures). We can reuse this knowledge.

Pre-trained network (ImageNet, 1,000 classes)
│
├── Early layers: general characteristics (edges, textures)
├── Middle layers: more complex shapes
└── Late shifts: task-specific characteristics

Transfer learning:
1. Load pre-trained network
2. Freeze the last layers
3. Add new classification header
4. Train only the head (few epochs)

12.2 Implementation with MobileNetV2

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# ── 1. Load base model (without classification header) ─────────────────────
base_model = keras.applications.MobileNetV2(
    input_shape=(96, 96, 3),
    include_top=False,          # Ohne letzten Dense-Schichten
    weights='imagenet'          # Vortrainierte Gewichte laden
)

# Freeze base model
base_model.trainable = False

# ── 2. Add new head ─────────────────────── ───────────────────────
inputs = keras.Input(shape=(96, 96, 3))
x = keras.applications.mobilenet_v2.preprocess_input(inputs)
x = base_model(x, training=False)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dropout(0.3)(x)
outputs = layers.Dense(10, activation='softmax')(x)   # 10 eigene Klassen

model = keras.Model(inputs, outputs)

# ── 3. Train only the head ────────────────────── ──────────────────────
model.compile(
    optimizer=keras.optimizers.Adam(0.001),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)
# model.fit(X_train, y_train, epochs=10, ...)

# ── 4. Fine-tuning: Release upper layers of the base model ────────────
base_model.trainable = True

# Only train the last 20 shifts
for layer in base_model.layers[:-20]:
    layer.trainable = False

model.compile(
    optimizer=keras.optimizers.Adam(1e-5),  # Sehr kleine Lernrate!
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)
# model.fit(X_train, y_train, epochs=5, ...)

12.3 When to use transfer learning?

Few own data + similar task → Feature Extraction (freeze base)
Few own data + different task → transfer learning + fine-tuning
Lots of your own data → fine-tuning or your own network

13. Summary and outlook

Overview

Deep learning
│
├── Basic building blocks
│ ├── Neuron: z = wᵀx + b, y = f(z)
│ ├── Activation: ReLU (standard), Sigmoid, Softmax
│ └── Loss: Cross-Entropy (Class.), MSE (Regr.)
│
├── Training
│ ├── Backpropagation + gradient descent
│ ├── Optimizer: Adam (default)
│ └── Regularization: Dropout, BatchNorm, Early Stopping
│
├── Architectures
│ ├── MLP / Dense: Tabular data
│ └── CNN: images, spatial data
│
└── Advanced
└── Transfer Learning: Reuse pre-trained models

Checklist for a deep learning project

✅ Normalize data (Pixel / 255 or StandardScaler)
✅ Select architecture for the task (images → CNN, otherwise MLP)
✅ Start with a simple model, then become more complex
✅ Use Dropout and BatchNorm to prevent overfitting
✅ Enable early stopping with restore_best_weights
✅ Visualize training history (loss + accuracy).
✅ If there is little data: consider transfer learning
✅ Use data augmentation for image data

Outlook: Next topics

• Natural Language Processing: Text preprocessing, embeddings, transformers
• Recurrent Networks (RNNs, LSTMs): Time series and sequences
• Generative Models: GANs, Variational Autoencoders
• MLOps: Version, deploy, and monitor models

Appendix: Quick Reference Keras

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# ── Build the model ──────────────────────────── ────────────────────────────
model = keras.Sequential([
    layers.Dense(128, activation='relu', input_shape=(n_features,)),
    layers.Dropout(0.3),
    layers.Dense(64, activation='relu'),
    layers.Dense(n_classes, activation='softmax')   # or 'sigmoid' for binary classification
])

# ── CNN block ─────────────────────────────── ───────────────────────────────
cnn_block = [
    layers.Conv2D(32, (3,3), activation='relu', padding='same'),
    layers.BatchNormalization(),
    layers.MaxPooling2D((2,2)),
    layers.Dropout(0.25),
]

# ── Compile ────────────────────────────── ──────────────────────────────
model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',  # Integer-Labels
    metrics=['accuracy']
)

# ── Train ────────────────────────────── ───────────────────────────────
history = model.fit(X_train, y_train,
                    epochs=50, batch_size=64,
                    validation_split=0.1,
                    callbacks=[
                        keras.callbacks.EarlyStopping(patience=5,
                            restore_best_weights=True)
                    ])

# ── Evaluate and predict ────────────────────── ───────────────────────
loss, acc = model.evaluate(X_test, y_test)
y_pred_proba = model.predict(X_test)
y_pred       = y_pred_proba.argmax(axis=1)

# ── Save/load model ─────────────────────── ────────────────────────
model.save('mein_modell.keras')
model = keras.models.load_model('mein_modell.keras')

HTL Pinkafeld – IF/IT | Data Science & AI | 12th grade