Natural Language Processing – Text Processing and Transformers

Subject: Data Science & Artificial Intelligence
School level: 12th grade – HTL Informatik
Requirements: Deep Learning, Python, NumPy, basic Keras
Author: HTL Pinkafeld – IF/IT

1. What is Natural Language Processing?

Natural Language Processing (NLP) is the branch of AI that deals with understanding, processing and generating human language.

1.1 Why is language difficult for computers?

"The bench is tough."          → Bank or financial institution?
"That doesn't happen easily."  → Idiom, not a literal meaning
"He saw her through the telescope."   → Who had the telescope?

Language is ambiguous, context-dependent, culturally influenced and constantly changing - all of which make it extremely demanding for computers.

1.2 Typical NLP tasks

1.3 NLP Pipeline Overview

Raw text
│
▼
Preprocessing (tokenization, stop words, stemming)
│
▼
Vectorization (text → numbers)
│
▼
Model (classic or neural)
│
▼
Output (class, sentiment, translation, ...)

2. Text preprocessing

Raw text needs to be cleaned and unified before modeling.

2.1 Tokenization

Tokenization breaks text into smaller units – tokens (words, punctuation marks or subwords):

# Simple word tokenization
text = "Deep Learning ist ein Teilgebiet der KI!"
tokens = text.lower().split()
print(tokens)
# ['deep', 'learning', 'is', 'a', 'subarea', 'the', 'ki!']

# Better: with NLTK
import nltk
nltk.download('punkt', quiet=True)
from nltk.tokenize import word_tokenize

tokens = word_tokenize(text.lower(), language='german')
print(tokens)
# ['deep', 'learning', 'is', 'a', 'subarea', 'the', 'ki', '!']

2.2 Remove stop words

Stop words are common words without any content contribution (e.g. “is”, “the”, “a”):

import nltk
nltk.download('stopwords', quiet=True)
from nltk.corpus import stopwords

stop_de = set(stopwords.words('german'))
tokens_clean = [t for t in tokens if t not in stop_de and t.isalpha()]
print(tokens_clean)
# ['deep', 'learning', 'subarea', 'ki']

2.3 Stemming and lemmatization

Both methods reduce words to their basic form:

from nltk.stem.snowball import GermanStemmer

stemmer = GermanStemmer()
woerter = ['lernend', 'gelernt', 'lernte', 'Lernen']
for w in woerter:
    print(f"{w:12} → {stemmer.stem(w)}")

2.4 Full preprocessing pipeline

import re
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.snowball import GermanStemmer

nltk.download(['punkt', 'stopwords'], quiet=True)

def preprocess(text, language='german', stem=False):
    """Complete text preprocessing."""
    # Lower case
    text = text.lower()
    # Remove special characters and numbers
    text = re.sub(r'[^a-z\s]', '', text)
    # Tokenize
    tokens = word_tokenize(text, language=language)
    # Remove stop words
    stop = set(stopwords.words(language))
    tokens = [t for t in tokens if t not in stop and len(t) > 2]
    # Optional: stemming
    if stem:
        stemmer = GermanStemmer()
        tokens = [stemmer.stem(t) for t in tokens]
    return tokens

# Example
texte = [
    "Das neue iPhone ist wirklich beeindruckend und sehr schnell!",
    "Dieses Produkt ist eine absolute Katastrophe, nie wieder!",
    "Ganz okay, nicht besonders gut aber auch nicht schlecht."
]

for t in texte:
    print(preprocess(t))

3. Texts as numbers – vectorization

Models work with numbers, not text. We need a representation that numerically captures the content of a text.

3.1 Bag of Words (BoW)

The text is displayed as a "bag of words" - only word frequencies count, the order is ignored:

from sklearn.feature_extraction.text import CountVectorizer

texte = [
    "das auto ist rot",
    "das fahrrad ist blau",
    "das auto und das fahrrad"
]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texte)

print("Vokabular:", vectorizer.get_feature_names_out())
print(X.toarray())
# Each line = one text, each column = one word
# Value = number of occurrences

Output:

Vocabulary: ['car' 'blue' 'the' 'bicycle' 'is' 'red' 'and']
[[1 0 1 0 1 1 0]
[0 1 1 1 1 0 0]
[1 0 2 1 0 0 1]]

Problem: Frequent words like “that” dominate, even though they say little.

3.2 TF-IDF

TF-IDF (Term Frequency – Inverse Document Frequency) weights words according to their informational relevance:

TF(t, d) = frequency of term t in document d
IDF(t) = log(N / df(t)) ... N = total number of documents, df = documents with term t

TF-IDF(t, d) = TF(t, d) × IDF(t)

• Frequent word in many texts → low IDF → low weight
• Rare word that occurs often in a text → high weighting

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split

# Example data set
texte = [
    "das auto ist sehr schnell",
    "ich liebe mein fahrrad",
    "das auto verbraucht viel benzin",
    "fahrrad fahren ist gesund",
    "bought a new car very cheaply",
    "mit dem fahrrad durch den park"
]
labels = [0, 1, 0, 1, 0, 1]   # 0 = Auto, 1 = Fahrrad

X_train, X_test, y_train, y_test = train_test_split(
    texte, labels, test_size=0.33, random_state=42
)

# Pipeline: TF-IDF + Naive Bayes (classic NLP baseline)
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=1000)),
    ('clf',   MultinomialNB())
])

pipeline.fit(X_train, y_train)
print(f"Accuracy: {pipeline.score(X_test, y_test):.2%}")

# New text prediction
neu = ["schnelles elektroauto gekauft"]
print(f"Klasse: {'Fahrrad' if pipeline.predict(neu)[0] else 'Auto'}")

3.3 Limits of classic vectorization

4. Word Embeddings

4.1 Basic idea

Word embeddings represent words as dense vectors in a low-dimensional space (e.g. 100-300 dimensions). Similar words have similar vectors.

"King" → [0.25, -0.12, 0.87, ...]
"Queen" → [0.23, -0.08, 0.84, ...] (similar!)
"Auto" → [-0.45, 0.33, -0.12, ...] (very different)

4.2 Word2Vec

Word2Vec (Google, 2013) learns embeddings from large amounts of text. The core idea: words that appear in similar contexts have similar meanings.

from gensim.models import Word2Vec

# Training data (sentences as token lists)
saetze = [
    ['the', 'king', 'rules', 'the', 'land'],
    ['die', 'koenigin', 'regiert', 'das', 'reich'],
    ['das', 'auto', 'faehrt', 'schnell'],
    ['das', 'fahrrad', 'faehrt', 'langsam'],
    ['der', 'hund', 'bellt', 'laut'],
    ['die', 'katze', 'schlaeft', 'viel'],
]

model = Word2Vec(saetze, vector_size=50, window=3, min_count=1, epochs=100)

# Find similar words
print(model.wv.most_similar('koenig', topn=3))

# Vector arithmetic: King - Man + Woman ≈ Queen
result = model.wv.most_similar(
    positive=['koenig', 'frau'],
    negative=['mann'],
    topn=1
)
print(f"King - Man + Woman ~= {result}")

4.3 The famous vector arithmetic

vec("King") - vec("Man") + vec("Woman") ≈ vec("Queen")
vec("Paris") - vec("France") + vec("Germany") ≈ vec("Berlin")
vec("run") - vec("run") + vec("swim") ≈ vec("swim")

This property shows that embeddings encode semantic relationships.

4.4 Keras Embedding Layer

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Use embedding layers directly in Keras
vocab_size    = 10000   # Vocabulary size
embedding_dim = 64      # Dimensionen des Embedding-Vektors
max_length    = 200     # Maximum text length (in tokens)

model = keras.Sequential([
    layers.Embedding(vocab_size, embedding_dim, input_length=max_length),
    # Output: (batch_size, max_length, embedding_dim)
    layers.GlobalAveragePooling1D(),   # Average over all tokens
    layers.Dense(64, activation='relu'),
    layers.Dense(1, activation='sigmoid')   # Binary classification
])

model.summary()

5. Recurrent Networks – RNN and LSTM

5.1 The problem with sequences

An MLP processes each input vector independently - it has no memory. For texts, however, the order of the words is crucial.

"The dog bites the man."    ≠ “The dog bites the man.”

5.2 Recurrent Neural Networks (RNN)

RNNs process sequences stepwise, passing a hidden state from one step to the next:

x₁ x₂ x₃ x₄
│ │ │ │
[RNN] → [RNN] → [RNN] → [RNN]
│ │ │ │
h₁ h₂ h₃ h₄ → output

hₜ = tanh(Wₕ hₜ₋₁ + Wₓ xₜ + b)

Problem: For long sequences, the RNN “forgets” early information (vanishing gradient over time).

5.3 Long Short-Term Memory (LSTM)

LSTM (Hochreiter & Schmidhuber, 1997) solves the forgetting problem through an explicit memory (cell state) and three gates that control what is stored, forgotten, or output:

┌─────────────────────────────────┐
│ Cell State Cₜ │
│ ┌──────────┐ ┌──────────┐ │
Forget ──►│ │ Forget │ │ Input │ │
Gate │ │ Gate │──►│ Gate │ │
Input ──►│ └──────────┘ └──────────┘ │
│ ↓ │
Output ◄──│ Output Gate │
└─────────────────────────────────┘

• Forget Gate: What should be deleted from memory?
• Input Gate: What should be saved?
• Output Gate: What should be output?

5.4 LSTM for sentiment analysis

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np

# ── Load data: IMDB film reviews ─────────────────── ────────────────────
vocab_size  = 10000
max_length  = 200
embed_dim   = 64

(X_train, y_train), (X_test, y_test) = keras.datasets.imdb.load_data(
    num_words=vocab_size
)

# Padding: make all sequences the same length
X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=max_length)
X_test  = keras.preprocessing.sequence.pad_sequences(X_test,  maxlen=max_length)

print(f"Trainingsdaten: {X_train.shape}")   # (25000, 200)

# ── LSTM model ───────────────────────────── ──────────────────────────────
model = keras.Sequential([
    layers.Embedding(vocab_size, embed_dim, input_length=max_length),
    layers.LSTM(64, return_sequences=True),   # Returns all hidden states
    layers.LSTM(32),                           # Nur letzten Hidden State
    layers.Dropout(0.5),
    layers.Dense(1, activation='sigmoid')     # Positiv / Negativ
])

model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['accuracy']
)

model.summary()

history = model.fit(
    X_train, y_train,
    epochs=5,
    batch_size=64,
    validation_split=0.1
)

test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"Test Accuracy: {test_acc:.2%}")   # Typisch: ~87%

5.5 RNN vs. LSTM vs. GRU

6. The Transformer Architecture

6.1 Motivation: Problems with RNNs

RNNs/LSTMs have two main weaknesses:

1. Sequential processing: Token by token - no parallelization
2. Limited memory: LSTMs also have problems with very long texts

The Transformer (Vaswani et al., 2017, “Attention is All You Need”) solves both problems.

6.2 Self-Attention – the heart of it

Self-Attention allows each token to pay attention to all other tokens in the sequence - simultaneously and in parallel.

Sentence: "The bench by the river is beautiful."

When processing "Bank":
→ high attention to “river” (context shows: bank, not financial institution)
→ high attention to “beautiful”
→ low attention on “Die”, “am”

Calculation (simplified):

Three vectors are calculated for each token:
Q (Query): "What am I looking for?"
K (Key): "What do I offer?"
V (Value): “What is my content?”

Attention(Q, K, V) = softmax(Q · Kᵀ / √dₖ) · V

dₖ = dimension of the key vectors (scaling for numerical stability)

6.3 Multi-Head Attention

Instead of a single attention calculation, the Transformer uses multiple parallel attention heads - each can focus on different relationships (e.g. grammatical structure, semantic similarity):

input
│
├─ Head 1: pays attention to syntactic relationships
├─ Head 2: pays attention to semantic relationships
├─ Head 3: pay attention to corefrence (which “he” is meant?)
└─ ...
│
Concatenation + linear projection
│
output

6.4 Complete Transformer architecture

Encoder (for text understanding): Decoder (for text generation):

Input tokens Output tokens (moved)
│ │
Positional Encoding Positional Encoding
│ │
┌─────────────┐ ┌─────────────────────┐
│ Multi-Head │ │ Masked Multi-Head │
│ Attention │ │ Attention │
├─────────────┤ ├─────────────────────┤
│ Add & Norm │ │ Cross Attention │
├─────────────┤ │ (on encoder) │
│ Feed-Forward│ ├─────────────────────┤
├─────────────┤ │ feed forward │
│ Add & Norm │ ├─────────────────────┤
└─────────────┘ × N └─────────────────────┘ × N
│ │
Encoder output Linear + Softmax
│
Next token

Positional Encoding: Since transformers do not know any order (unlike RNNs), position information is explicitly added to the embedding as vectors.

7. BERT and pre-trained language models

7.1 What is BERT?

BERT (Bidirectional Encoder Representations from Transformers, Google 2018) is a pre-trained transformer trained on massive amounts of text.

Two training tasks:

• Masked Language Model: 15% of tokens are masked, BERT needs to predict them
• Next Sentence Prediction: Do two sentences belong together?

Input: "The [MASK] is sleeping on the sofa."
BERT: "The dog is sleeping on the sofa."   (high confidence)
"The table sleeps on the sofa."  (low confidence)

7.2 BERT is bidirectional

In contrast to GPT (left→right), BERT reads both directions at the same time:

GPT (unidirectional): The → dog → barks → ?
BERT (bidirectional): ← The [MASK] barks → (sees context left AND right)

This makes BERT particularly good for comprehension tasks (classification, NER, QA).

7.3 Important pre-trained models

7.4 Hugging Face – the NLP library

Hugging Face Transformers is the central library for pre-trained models:

pip install transformers datasets

from transformers import pipeline

# Sentiment analysis (English, without your own training!)
classifier = pipeline('sentiment-analysis')
results = classifier([
    "This product is absolutely amazing!",
    "I'm really disappointed with this purchase.",
    "It's okay, nothing special."
])

for r in results:
    print(f"{r['label']:10} ({r['score']:.2%})")

# Multilingual: Model for German
de_classifier = pipeline(
    'sentiment-analysis',
    model='oliverguhr/german-sentiment-bert'
)
print(de_classifier("Das Produkt ist fantastisch!"))

8. Practical Project: Sentiment Analysis

8.1 Task

Product Review Analysis: Is the sentiment positive, negative or neutral?

8.2 Full pipeline with TF-IDF + classic ML

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import classification_report, confusion_matrix
import seaborn as sns
import matplotlib.pyplot as plt
import re

# ── 1. Create data ────────────────────────── ──────────────────────────
bewertungen = [
    # Positive (Label 2)
    ("Absolutely thrilled! Top quality and fast delivery.", 2),
    ("Sehr zufrieden, genau wie beschrieben. Klare Kaufempfehlung!", 2),
    ("Perfect product, does exactly what it promises. 5 stars.", 2),
    ("Super schnelle Lieferung, Produkt ist fantastisch.", 2),
    ("Bin total begeistert, werde es weiterempfehlen.", 2),
    ("Outstanding quality, gladly again.", 2),
    # Neutral (Label 1)
    ("Produkt ist okay, nichts Besonderes aber funktioniert.", 1),
    ("Okay for the price. Not bad, not great.", 1),
    ("Expected more, but it is okay. Mediocre.", 1),
    ("Average. I would neither recommend nor discourage it.", 1),
    # Negative (Label 0)
    ("Total disappointment! The quality is catastrophic.", 0),
    ("Nie wieder! Das Produkt ist nach einer Woche kaputt.", 0),
    ("Absolut furchtbar. Geld raus geworfen.", 0),
    ("Does not work at all as advertised. Returning it!", 0),
    ("Terrible quality, completely disappointed.", 0),
]

df = pd.DataFrame(bewertungen, columns=['text', 'label'])
label_names = {0: 'Negativ', 1: 'Neutral', 2: 'Positiv'}

X_train, X_test, y_train, y_test = train_test_split(
    df['text'], df['label'], test_size=0.25, random_state=42, stratify=df['label']
)

# ── 2. Build pipeline ───────────────────────── ─────────────────────────
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(
        max_features=500,
        ngram_range=(1, 2),      # Unigramme + Bigramme
        sublinear_tf=True        # log(TF) instead of TF
    )),
    ('clf', LogisticRegression(max_iter=1000, C=1.0))
])

pipeline.fit(X_train, y_train)

# ── 3. Evaluation ──────────────────────────── ────────────────────────────
y_pred = pipeline.predict(X_test)
print(classification_report(y_test, y_pred,
      target_names=[label_names[i] for i in sorted(label_names)]))

# ── 4. Confusion Matrix ───────────────────────── ──────────────────────────
cm = confusion_matrix(y_test, y_pred)
sns.heatmap(cm, annot=True, fmt='d', cmap='Oranges',
            xticklabels=label_names.values(),
            yticklabels=label_names.values())
plt.title('Confusion Matrix – Sentiment-Analyse')
plt.tight_layout()
plt.show()

# ── 5. Classify new reviews ────────────────────────────────────
neue_texte = [
    "Ich bin sehr zufrieden mit dem Kauf!",
    "The device broke after 3 days.",
    "Naja, geht so. War billiger als gedacht."
]

for text in neue_texte:
    pred = pipeline.predict([text])[0]
    proba = pipeline.predict_proba([text])[0]
    print(f"{label_names[pred]:10} ({proba.max():.0%}) | {text}")

8.3 Show important features

# Which words are the most meaningful?
feature_names = pipeline['tfidf'].get_feature_names_out()
coef          = pipeline['clf'].coef_

fig, axes = plt.subplots(1, 3, figsize=(14, 5))
klassen = ['Negativ', 'Neutral', 'Positiv']

for i, (ax, klasse) in enumerate(zip(axes, klassen)):
    top_idx = coef[i].argsort()[-10:][::-1]
    top_feat = feature_names[top_idx]
    top_coef = coef[i][top_idx]
    ax.barh(top_feat, top_coef, color='#F6A01D')
    ax.set_title(f'Top Features: {klasse}')
    ax.invert_yaxis()

plt.tight_layout()
plt.show()

9. Practical Project: Text Classification with BERT

9.1 Fine-tuning a German BERT model

from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
    TrainingArguments,
    Trainer
)
from datasets import Dataset
import torch
import numpy as np

# ── 1. Prepare data ───────────────────────── ─────────────────────────
texte = [
    "The product is fantastic and does what it promises.",
    "Absolutely disappointing, never again!",
    "Quite okay for the price.",
    "Excellent quality, highly recommended.",
    "Katastrophale Verarbeitung, sofort kaputt gegangen.",
    "Durchschnittlich, nichts Besonderes.",
]
labels = [2, 0, 1, 2, 0, 1]   # 0=Negativ, 1=Neutral, 2=Positiv

dataset = Dataset.from_dict({'text': texte, 'label': labels})
dataset = dataset.train_test_split(test_size=0.33)

# ── 2. Load tokenizer ────────────────────────── ──────────────────────────
model_name = 'deepset/gbert-base'   # Deutsches BERT
tokenizer  = AutoTokenizer.from_pretrained(model_name)

def tokenize(batch):
    return tokenizer(batch['text'], truncation=True,
                     padding='max_length', max_length=128)

dataset = dataset.map(tokenize, batched=True)

# ── 3. Load and adjust model ───────────────────── ─────────────────────
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=3   # Negativ, Neutral, Positiv
)

# ── 4. Training ───────────────────────────── ──────────────────────────────
training_args = TrainingArguments(
    output_dir='./bert_sentiment',
    num_train_epochs=3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    evaluation_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
    logging_steps=10,
)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=1)
    return {'accuracy': (preds == labels).mean()}

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset['train'],
    eval_dataset=dataset['test'],
    compute_metrics=compute_metrics,
)

trainer.train()

# ── 5. Prediction ──────────────────────────── ─────────────────────────────
def predict_sentiment(text):
    inputs = tokenizer(text, return_tensors='pt',
                       truncation=True, max_length=128)
    with torch.no_grad():
        outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=1)[0]
    label = probs.argmax().item()
    names = ['Negativ', 'Neutral', 'Positiv']
    return f"{names[label]} ({probs[label]:.1%})"

print(predict_sentiment("Das Produkt ist wirklich toll!"))
print(predict_sentiment("I am very disappointed with it."))

10. Large language models and ChatGPT

10.1 What are Large Language Models (LLMs)?

LLMs are transformer models with billions of parameters trained on massive amounts of text:

10.2 How does ChatGPT work?

1. Pretraining (GPT basis):
Predict next token on huge amounts of text
→ Model learns language, facts, connections

2. Supervised Fine Tuning (SFT):
Human demonstrations of good answers
→ Model learns conversation format

3. RLHF (Reinforcement Learning from Human Feedback):
People evaluate answers → reward model
→ Model is optimized for human preferences

10.3 Prompt Engineering

The quality of the output depends heavily on the prompt:

# Bad prompt:
"Explain ML"

# Better prompt (with context and format):
"""
You are an experienced computer science teacher at a technical high school.
Explain machine learning for 16-year-old students with Python knowledge.
Use a simple analogy and keep the explanation to 3-4 sentences.
"""

# Few-Shot Prompting (give examples):
"""
Klassifiziere die Stimmung als POSITIV, NEGATIV oder NEUTRAL:
Example: "That was great!" -> POSITIVE
Example: "Never again." -> NEGATIVE
Example: "So-so." -> NEUTRAL

Jetzt: "Bin wirklich beeindruckt!" →
"""

10.4 Anthropic API in Python

import anthropic

client = anthropic.Anthropic(api_key="dein-api-key")

message = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Explain the difference between RNN and Transformer."
        }
    ]
)

print(message.content[0].text)

11. Ethical aspects in NLP

11.1 Bias in language models

Language models learn from human texts - and in the process also adopt human prejudices:

# Classic example of bias in Word2Vec:
# vec("Doctor") - vec("Man") + vec("Woman") ≈ vec("Nurse")
# (not vec("doctor")!)

Causes:

• Training data reflects social inequalities
• Some groups are underrepresented in texts
• Historical texts contain outdated worldviews

11.2 Hallucination

LLMs can make high confidence statements that are factually false:

User: "When was HTL Pinkafeld founded?"
LLM: "The HTL Pinkafeld was founded in 1923." (formulated incorrectly, but convincingly)

Countermeasures: Retrieval-Augmented Generation (RAG) – Model first searches trustworthy sources before responding.

11.3 Data Protection

• Training data may contain personal information
• In extreme cases, models can “memorize” training data
• Input into external APIs (e.g. ChatGPT) should not contain sensitive data

11.4 Responsible use

✅ Critically question sources
✅ Label AI-generated content as such
✅ Cross-check hallucinations with trustworthy sources
✅ Do not enter sensitive data in external models
✅ Perform bias testing on critical applications

12. Summary and outlook

NLP learning path

Classic NLP Modern NLP
────────────── ────────────
Tokenization Word Embeddings (Word2Vec)
Stop words │
BoW/TF-IDF RNN/LSTM
│ │
Naive Bayes Transformers
Logistic Regression (Self-Attention)
│
BERT/GPT/T5
(pre-trained + fine-tuned)
│
ChatGPT / Claude
(RLHF, instruction tuning)

Decision support

Task known, little data?
→ Pre-trained model (Hugging Face Pipeline)

Simple classification, lots of data?
→ TF-IDF + logistic regression as baseline

Complex classification?
→ BERT fine tuning

Text generation?
→ GPT based model or LLM API

Pattern recognition in sequences?
→ LSTM or Transformer

Checklist

✅ Always preprocess text (lower, clean, tokenize)
✅ Start with a simple baseline (TF-IDF + LogReg)
✅ Prefer pre-trained models (transfer learning!)
✅ Choose a tokenizer that fits the model
✅ Test for bias and hallucinations
✅ Evaluate results with classification_report

Outlook: Next topics

• MLOps: Version, deploy, and monitor models
• Retrieval-Augmented Generation (RAG): Expand LLMs with your own knowledge
• Multimodal models: Text + Image (GPT-4o, Gemini)

Appendix: Quick Reference NLP

# ── Text preprocessing ───────────────────────── ──────────────────────────
import re, nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
nltk.download(['punkt', 'stopwords'], quiet=True)

def clean(text, lang='german'):
    text = text.lower()
    text = re.sub(r'[^a-z\s]', '', text)
    tokens = word_tokenize(text, language=lang)
    stop = set(stopwords.words(lang))
    return [t for t in tokens if t not in stop and len(t) > 2]

# ── TF-IDF + classification ─────────────────────── ────────────────────────
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=5000, ngram_range=(1,2))),
    ('clf',   LogisticRegression(max_iter=1000))
])
pipeline.fit(X_train, y_train)
pipeline.predict(["neuer Text"])

# ── Hugging Face Pipeline (simplest variant) ───────────────────────────
from transformers import pipeline as hf_pipeline

nlp = hf_pipeline('sentiment-analysis',
                   model='oliverguhr/german-sentiment-bert')
nlp(["Great product!", "Very disappointing."])

# ── IMDB LSTM ────────────────────────────── ───────────────────────────────
from tensorflow import keras
from tensorflow.keras import layers

(X_train, y_train), (X_test, y_test) = keras.datasets.imdb.load_data(num_words=10000)
X_train = keras.preprocessing.sequence.pad_sequences(X_train, maxlen=200)

model = keras.Sequential([
    layers.Embedding(10000, 64, input_length=200),
    layers.LSTM(64),
    layers.Dense(1, activation='sigmoid')
])
model.compile('adam', 'binary_crossentropy', ['accuracy'])

HTL Pinkafeld – IF/IT | Data Science & AI | 12th grade