The GenAI Threat Model
The GenAI threat model is the catalog of ways a language-model system can be turned against its owner, and it looks nothing like classic application security. The distinguishing property is that the attack surface is the input itself: any text the model reads, a user message, a retrieved document, a webpage fetched by a tool, can carry instructions. The OWASP LLM Top 10 is the standard map: prompt injection, insecure output handling, training data poisoning, sensitive information disclosure, and excessive agency cover most real incidents. The highest-consequence pattern combines three of them: an injected instruction persuades an agent with tool access to exfiltrate data through a channel the network team never modeled, such as a markdown image URL or an outbound API call. The architect's job is to assume the model will eventually comply with a hostile instruction and to design the blast radius accordingly: least-privilege tools, egress controls, and human approval on irreversible actions.
Prompt Injection & Jailbreaks
Prompt injection is the exploitation of a model's inability to reliably separate instructions from data: everything in the context window is just tokens, and the model weighs them all. Direct injection (a user typing "ignore previous instructions") is the tame variant; the dangerous one is indirect injection, where hostile instructions arrive inside content the system fetched on its own, a retrieved PDF, an email being summarized, a scraped webpage. Jailbreaks are the adjacent craft of defeating safety training through role-play framing, encoding tricks, or many-shot patterns. Mitigations are layered, not absolute: privilege separation between trusted system prompts and untrusted content, output mediation so model responses cannot directly trigger consequential actions, least-privilege tool scopes, and classifiers screening inputs and outputs. The honest architectural position is that no complete fix exists today, so the design question is not whether injection succeeds but what a successful injection is able to reach.
The EU AI Act, an Architect's Guide
The EU AI Act is the first comprehensive AI statute, and it regulates by risk tier: prohibited practices (social scoring, manipulative systems), high-risk systems (hiring, credit, critical infrastructure), limited-risk transparency duties, and a separate track for general-purpose models. The timeline is staged: the Act entered into force in August 2024, prohibitions applied from February 2025, GPAI obligations from August 2025, and high-risk requirements phase in through 2026 and 2027. For an architect, the operative duties are concrete: automatic event logging with retention, human oversight designed into the workflow rather than bolted on, technical documentation covering training data provenance and evaluation results, and post-market monitoring. The practical trap is classification: a chatbot that drifts into employment screening changes tier, and with it the compliance bill. Build the logging, documentation, and oversight hooks into the platform layer once, so each new use case inherits them instead of relitigating compliance per project.
NIST AI RMF & AI TRiSM, Operationalized
NIST's AI Risk Management Framework and Gartner's AI TRiSM are the two vocabularies most enterprise AI governance programs get built from, and both are useless until translated into controls. The AI RMF organizes work into four functions: Govern (policy, accountability, risk appetite), Map (context and impact per use case), Measure (evaluation, testing, tracked metrics), and Manage (prioritization, response, decommissioning). TRiSM slices the same problem by layer: explainability and model monitoring, ModelOps, AI application security, and privacy. Operationalizing them means each abstract statement acquires an owner, a system, and evidence: "Measure" becomes a mandatory evaluation gate in the deployment pipeline; "Govern" becomes an AI inventory with a risk tier per entry and a review board with real veto power. The failure mode is a framework mapping exercise that produces a spreadsheet and no enforcement; the test is whether a noncompliant use case is actually blocked from shipping.
Privacy, PII & Data Residency
Privacy in GenAI systems is mostly a data-flow problem: prompts, retrieved context, and completions all carry personal data through infrastructure that logging, caching, and vendor telemetry can silently retain. The working control set: PII detection and redaction at the gateway (NER-based tools like Presidio, or an LLM classifier pass) with reversible tokenization when downstream systems need to re-identify; contractual and technical guarantees on training-data rights, meaning zero-retention API terms and opt-outs verified rather than assumed; retention schedules for prompt logs, which are discovery-relevant records, not debug exhaust; and data residency, pinning inference and storage to a region, which constrains model choice since not every frontier model is served from every geography. Under GDPR, most non-trivial deployments warrant a DPIA. The architect's trade-off is precision versus utility: aggressive redaction degrades answers on exactly the customer-service and healthcare use cases where personal context is the point, so redaction policy should be per use case, not global.
Model Risk Management
Model risk management is the discipline banks built under the Federal Reserve's SR 11-7 guidance, now being retrofitted for GenAI: every model gets an inventory entry, an owner, independent validation before use, and ongoing monitoring. The GenAI adaptation is real work because the classic toolkit assumes a fixed model with a defined input space, and a foundation model behind an API is neither: the vendor can revise weights under you, and the input space is all of language. What transfers: a mandatory model inventory (including "hidden" models inside SaaS products), model cards documenting intended use, evaluation results and known failure modes, validation as challenge by a team independent of the builders, and drift monitoring that tracks behavioral evals over time rather than input distributions alone. The architect's decision is proportionality: tiering validation depth by materiality, since applying full banking-grade validation to an internal meeting summarizer stalls the portfolio for no risk reduction.
Red-Teaming & Safety Testing
Red-teaming is adversarial testing that treats your own AI system the way an attacker or a hostile user would, before they get the chance. A working program combines automated jailbreak suites (tools like garak or PyRIT running thousands of known attack patterns per release), manual expert probing for novel failure modes, and domain-specific harm scenarios written with the business: wrong-dosage answers for a pharma assistant, fabricated fee waivers for a banking bot, discriminatory screening for an HR tool. Findings feed the evaluation set, so every discovered exploit becomes a permanent regression test. The defining property is that this is continuous, not a pre-launch ceremony: model upgrades, prompt changes, and new tools all reset the risk surface, and attackers iterate weekly. The architect's decision is scope and cadence: automated suites on every release, expert red-teams quarterly or on major changes, and a severity taxonomy agreed with risk owners so findings actually block launches instead of decorating a report.