Resilient AI: Securing Large Language Models

Sep 3, 2026, 9:00 AM → Sep 4, 2026, 4:00 PM Europe/Berlin

ScaDS.AI Dresden/Leipzig, Strehlener Str. 14, 01069 Dresden

Motivation

Large Language Models (LLMs) are increasingly deployed in research and real-world applications, where they interact with sensitive data, external tools, and human users. Despite their capabilities, LLMs introduce novel security and privacy challenges, including prompt injection, data leakage, and unintended disclosure of sensitive information. These risks stem from the fundamental design of LLMs, which interpret natural language instructions without reliable mechanisms to distinguish trusted from adversarial inputs. As a result, ensuring the resilience, robustness, and trustworthiness of LLM-based systems has become a critical research challenge.

This workshop aims to provide an accessible overview of how LLMs work, introduce common attack scenarios, discuss privacy risks, and present methods for adding safeguards and protective mechanisms when building LLM-based systems.

Workshop Goals

Participants will:

• Understand how LLMs work (internals and architecture)
• Understand security, privacy, and reliability risks
• Perform hands-on attacks (prompt injection, data leakage)
• Learn how to design secure, privacy-aware LLM systems

Target Audience

• General audience: technical insights into LLMs and their vulnerability
• Developers, IT professionals: hands-on building secure LLM apps

Thu, September 3

1 Session 1: Introduction to LLMs

Motivates LLM security through real-world incidents, then gives an accessible introduction to LLM internals: tokenization, context windows, the transformer architecture, training stages (pre-training, fine-tuning, RLHF), prompt roles, and how generation works.

Speaker: Prof. Simon Razniewski (TU Dresden)

2 Session 2: LLM Weaknesses & Hacking

Introduces the main attack categories: prompt injection, jailbreaking techniques, training data extraction, hallucination, and supply-chain risks such as system prompt theft and model poisoning.

Speaker: Dr Himanshu Beniwal (TU Dresden)

12:00 PM Lunch Break

3 Hands-on: Breaking an LLM

Participants attack a pre-built chatbot through staged challenges—role-play attacks, delimiter manipulation, and indirect injection—aiming to extract the hidden system prompt or bypass safety checks.

Speakers: Dr Paramita Mirza (TU Dresden), Dr Siavash Ghiasvand (TU Dresden)

4 Results & Debrief

The group compares which attacks succeeded and why, with discussion of what participants would do differently as developers.

Fri, September 4

5 Session 3: LLMs & Privacy

Examines how LLMs expose sensitive information: training data memorization, PII leakage, differential privacy and its limits, GDPR requirements, and re-identification attacks.

Speaker: Dr Hermann Diebel-Fischer (TU Dresden)

6 Session 4: Guardrails & Defense Mechanisms

Surveys defensive tools—input validation, output filtering, alignment techniques, LLM-as-judge, and prompt firewalls—along with their limitations.

7 Session 5: Secure LLM Architecture & Design

Covers designing secure LLM systems: least privilege for agents, trust boundaries, access control for RAG, logging and auditing, and STRIDE-adapted threat modelling.

12:00 PM Lunch Break

8 Hands-on: Break & Fix a RAG System

Participants exploit a deliberately vulnerable RAG pipeline, then fix it with access control and output filtering, documenting their work in a short remediation note.

Speakers: Dr Paramita Mirza (TU Dresden), Dr Siavash Ghiasvand (TU Dresden)

9 Results & Wrap-up

Groups present findings, facilitators distill a practical pre-shipping checklist, and the workshop closes with further reading and next steps.