System Prompt Design Patterns
A system prompt is the standing contract between your application and the model: it fixes role, constraints, refusal behavior and output format before any user input arrives. Well-built ones follow a consistent anatomy: identity and role first, then hard constraints and refusal rules, then the output contract (schema, tone, length), with the most load-bearing instructions stated early and restated near the end, since models weight both positions more heavily than the middle. Specificity beats volume: "answer only from the provided context; if it is absent, say so" outperforms three paragraphs of vague guidance. Basic injection resistance belongs here too: delimit untrusted input clearly, state that user content cannot override the system role, and never place secrets in the prompt on the assumption they stay hidden, because they do not. The architectural point: the system prompt is production configuration. Treat an unreviewed edit to it as seriously as an unreviewed schema migration.
Structured Outputs & Schema Enforcement
Structured output turns a model from a text generator into a system component that downstream code can trust. There are two enforcement layers. At decode time, constrained decoding (JSON mode, grammar-based sampling, or provider-native schema enforcement) masks invalid tokens so the model literally cannot emit malformed JSON; this guarantees syntax but not semantics, since a syntactically perfect object can still carry a hallucinated enum value or an implausible amount. The second layer is validation plus retry: parse the response against a strict schema (Pydantic, Zod, plain JSON Schema), and on failure re-prompt with the validation error appended, which in practice resolves the large majority of failures within one or two retries. Field descriptions in the schema act as micro-prompts, so write them as deliberately as the prompt itself. The trade-off to price in: strict decoding constraints can degrade reasoning quality on hard tasks, so many teams let the model reason in free text first and emit the object last.
Few-Shot: When Examples Beat Instructions
Few-shot prompting exploits in-context learning: place worked examples in the prompt and the model infers the task from them, often more faithfully than from any instruction you could write. It earns its keep where the rules are hard to articulate but easy to demonstrate: idiosyncratic classification taxonomies, domain-specific extraction, tone-matched drafting. Selection matters far more than count; two or three examples covering the tricky boundary cases beat ten easy ones, and dynamically retrieving the most similar examples per query via embedding similarity is one of the highest-leverage upgrades in applied prompting. Format bleeds into behavior: models copy the label casing, delimiter style, and even the length distribution of your examples, so a sloppy example set produces sloppy output. The risks are concrete: every example consumes token budget on every call, examples can silently pin outdated policy, and an unbalanced label distribution biases classification. Curate the set like test data, because functionally it is.
Context Management: Compression, Budgets & Rot
Context management is the discipline of deciding what earns a place in the model's window on each call, because everything you include competes with everything else for attention. Mature systems assign explicit token budgets per component: system prompt, tool definitions, retrieved documents, conversation history, scratch state, and they enforce those budgets in code rather than hoping totals stay small. Long sessions suffer context rot: accumulated tool outputs, stale intermediate reasoning and repeated boilerplate dilute the signal, and models measurably lose recall for material buried mid-window even when the window nominally holds hundreds of thousands of tokens. The countermeasures are unglamorous and effective: summarize completed conversation segments into compact state, prune tool results once consumed, deduplicate retrieved passages, and rank the survivors by relevance to the current turn. The economics point the same direction: input tokens dominate cost in most agentic workloads, so context discipline doubles as a cost-control program.
Memory Architectures: Session, Episodic, Long-Term
Memory in an LLM system is not one thing; it is at least three tiers with different storage and lifecycle rules. Session memory is the live context window: fast, exact, and gone when the conversation ends or overflows. Episodic memory persists summaries of past interactions (what happened with this customer last Tuesday) in a store keyed by user and time, retrieved when relevant. Long-term memory holds durable facts and preferences as compact structured records, often a simple document per user loaded at every session start. Both persistent tiers are retrieval-backed: writes pass through an extraction step that decides what is worth keeping, and reads pass through relevance filtering so old memories do not crowd out the current task. The hard design decisions are the write policy (what qualifies, whether the user confirms it) and forgetting: TTLs, user-initiated deletion, and contradiction resolution when new facts supersede old ones. Under GDPR, memory is personal data; architect deletion in from day one.
Query Rewriting & Input Shaping
Query rewriting inserts a transformation step between what the user typed and what the system actually executes, because raw user input is usually a poor retrieval key. The standard moves: canonicalization resolves pronouns and ellipsis against conversation history, so "what about the second one?" becomes a self-contained question; decomposition splits multi-part questions into sub-queries that are retrieved and answered independently, then composed; and hypothetical-document expansion (HyDE) has the model draft a plausible answer first and embeds that draft, which lands closer to relevant passages in vector space than a terse question does. A small, fast model typically handles rewriting, adding a few hundred milliseconds and negligible cost while lifting retrieval hit rates substantially. The failure mode to guard against is over-rewriting: a rewriter that hallucinates intent silently answers a question the user never asked. Log the rewritten query alongside the original, and evaluate the rewriter as its own component.
Prompts as Managed Assets: Versioning & Testing
Prompts change system behavior the way code changes application behavior, so mature teams manage them with the same machinery. That means prompts live in version control or a registry with immutable versions, never in a dashboard textbox someone edits at six on a Friday; every change carries a diff, an author and a review. The regression suite is an eval set: a few hundred representative inputs with graded expected behaviors, scored automatically (exact match, programmatic checks, rubric-based LLM judges) on every prompt or model change, because a one-word edit can fix ten cases and quietly break thirty others. Rollout is staged like any deploy: shadow traffic, a canary percentage, comparison of quality scores and cost, then promotion, with instant rollback to a pinned version. The organizational shift matters most: once prompts are versioned, tested artifacts with named owners, prompt engineering stops being tribal knowledge and becomes an auditable engineering practice.