Kursplan
Module 1 — How AI Apps Break
Lab: None — architecture walkthrough & discussion
A builder’s mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from the developer’s perspective
- The request/response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Where untrusted data enters (and re-enters) the model
- Trust boundaries a developer owns vs. inherits
- Why AI attacks are semantic, not syntactic
- Mapping the OWASP LLM Top 10 to code you write
Key insight: Every point where untrusted text reaches the model — or where model output reaches your code — represents a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The “SQL injection moment” for AI — but you cannot fully escape it.
Topics:
- Direct vs. indirect prompt injection
- Hidden instructions in documents, web pages, and tool outputs
- Jailbreaks and role-confusion
- Why separating instructions from data matters
- Defensive prompt design (delimiters, structure, minimal authority)
- Why prevention is partial — design for containment
Hands-on:
- Attack your own chatbot
- Bypass a naive filter
- Restructure the prompt to shrink the blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The bug class developers underestimate most.
Topics:
- Model output as untrusted input to the rest of the app
- Insecure output handling (LLM02): XSS, SSRF, command/SQL injection downstream
- Never eval/exec/render raw model output
- Structured outputs and schema validation
- Output encoding and allowlists
- Safe rendering in web/UI contexts
Hands-on:
- Find and fix an insecure-output-handling vulnerability
- Enforce a JSON schema on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
One of the biggest new attack surfaces — and it’s yours to build.
Topics:
- Vector DB and retrieval threats
- Ingestion sanitization
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions in retrieved content (indirect injection)
- Data exfiltration via retrieval
Hands-on: Poison a RAG pipeline with a malicious document – add ingestion sanitization and retrieval scoping to defend it.
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
Where a bug becomes an action.
Topics:
- Excessive agency (LLM06) and tool abuse
- Least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop
- Sandboxing tool execution
- Scoped, short-lived credentials for agents
- Limiting autonomous loops and chaining
Hands-on:
- Lock down an over-permissioned agent
- Add an allowlist + approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost
Lab: Lab 05 — 05-Secrets-and-Cost
The operational mistakes that hurt fastest.
Topics:
- API key and secret management (never in prompts, code, or logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval
- Denial-of-wallet: unbounded token/cost consumption
- Rate limits, token budgets, and timeouts
- Logging without leaking secrets or PII
Hands-on:
- Move secrets out of the prompt/code path
- Add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Buy vs. build for input/output safety.
Topics:
- What guardrail frameworks do (and don’t)
- Input guardrails: injection/PII/topic classifiers
- Output guardrails: validation, filtering, grounding checks
- When a guardrail is appropriate vs. your own deterministic check
- Layering guardrails with the controls from earlier modules
- Performance, false positives, and failure modes
Hands-on:
- Add an input/output guardrail layer to an AI feature
- Measure what it catches and what it misses
Module 8 — Red-Teaming Your Own App
Lab: Lab 07 — 07-Red-Teaming
Ship it like an attacker already has it.
Topics:
- Building an abuse/test suite for AI features
- Automated prompt-injection and jailbreak tests
- Regression-testing guardrails and policies
- Running AI security checks in CI
- Model and dependency supply chain (provenance, pinning)
- A pre-ship security checklist for AI features
Hands-on:
- Write automated red-team tests for an AI feature
- Wire them into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: None — scoring exercise (uses the Capstone app)
Turn everything you’ve built into a repeatable score.
Topics:
- The AI Security Hexagon: six questions instead of “is it secure?”
- The six scored categories (Data, Prompt, Agent, Supply Chain, Detection, Governance)
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Scoring before/after hardening as a metric
Hands-on:
- Score the Capstone app on the 100-point scale
- Name the single change that most raises the score
Key insight: The three highest-weighted categories map to the trust boundaries a developer owns — so the score measures exactly what this course taught.
Capstone
Students harden a deliberately vulnerable AI application end-to-end.
The starter app contains:
- An injectable prompt
- Insecure output handling
- An unscoped RAG pipeline
- An over-permissioned agent
- Secrets in the prompt path
- No cost limits
Students apply the course:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval
- Apply least privilege and approval gates to the agent
- Move secrets out and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened app plus a short OWASP LLM Top 10 self-assessment.
Module - Lab map
Labs run in lab order, which follows module order. The course has 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion and Module 9 is a scoring exercise, so neither has its own lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab — it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it runs as a scoring exercise against the Capstone app.
Krav
- Ferdighetsnivå: Mellom.
- Studentene bør være komfortable med: bygging og konsumering av REST-API-er, et skriptspråk (lab-bruker Python), grunnleggende applikasjonsautentisering, git og CLI.
- Ingen maskinlæringsbakgrunn kreves — dette er et applikasjonssikkerhetskurs for personer som bygger med LLMs, ikke de som trener dem.
Målgruppe
- Programvare- / backend-ingeniører som bygger LLM-funksjoner
- Full-stack og API-utviklere
- AI/ML-applikasjonsingeniører
- Plattformingeniører som leverer copiloter og agenter
- Tekniske ledere og senioringeniører som eier AI-funksjoner
Referanser (2)
Jeg syntes virkelig det var interessant å lære om AI-angrep og verktøyene som finnes for å starte med øving og aktiv bruk i sikkerhetstesting. Jeg tok med meg mye kunnskap som jeg ikke hadde fra starten, og kurset innfridde det jeg håpet det skulle. Min favorittdel fra opplæringen var Comet Browser, og jeg var imponert over hva det kunne gjøre. Det er noe jeg helt sikkert vil se nærmere på. Til sammen var det en fantastisk kurs, og jeg trivdes med å lære om OWASP GenAI Top 10.
Patrick Collins - Optum
Kurs - OWASP GenAI Security
Maskinoversatt
Den profesjonelle kunnskapen og måten han presenterte den for oss
Miroslav Nachev - PUBLIC COURSE
Kurs - Cybersecurity in AI Systems
Maskinoversatt
