Anatomy of an LLM API
Nearly every model you will run in production is fronted by the same interface: a stateless chat completions endpoint that accepts an array of messages and returns, or streams, a response. The roles carry the contract: system sets behavior, user and assistant carry the conversation, tool returns the results of function calls the model requested. Streaming runs over server-sent events: the API emits incremental token deltas, so a thirty-second generation still shows sub-second time to first token. Tool calling is the load-bearing feature for agents: you pass JSON Schema definitions, the model emits structured call requests, and your runtime executes them and appends the results. Every response carries a usage block with prompt and completion token counts, and that one field is your metering, chargeback, and unit-cost model. The OpenAI-compatible shape is now the de facto standard (vLLM, NIM, and most gateways speak it), which is what makes provider portability a realistic architectural goal rather than a slide.
Platform Landscape: Foundry, Bedrock, Vertex, NIM & Self-Hosted
The managed platforms, Azure AI Foundry, AWS Bedrock, and Google Vertex AI, all solve the same problem: frontier and open-weight models served behind enterprise controls, so the models themselves rarely decide the choice. What actually differentiates them: which frontier family you get first-party (OpenAI models on Foundry, Anthropic's Claude on Bedrock and Vertex, Gemini on Vertex), how cleanly model access inherits your existing IAM, network perimeter, and data residency commitments, and the economics of provisioned versus on-demand throughput. NVIDIA NIM packages models as optimized containers, TensorRT-LLM behind an OpenAI-compatible endpoint, for teams that must run inference on GPUs they control, and self-hosted vLLM reaches the same outcome at the cost of owning the serving stack: batching, KV-cache management, upgrades, capacity planning. The honest selection criterion is usually gravity: enterprises pick their incumbent cloud's platform, then keep a gateway in front of it so the second source stays credible.
Model Gateways & AI Middleware
A model gateway is a reverse proxy built for LLM traffic: one endpoint, one credential model, one policy surface in front of every provider you use. Mechanically it terminates authentication and maps enterprise identities to virtual keys, enforces per-team quotas and rate limits, routes by model alias so claude-sonnet resolves to whichever deployment you choose, retries and fails over across regions and providers, caches responses (exact-match or semantic), and emits per-request telemetry: tokens, latency, cost, caller. Open-source options such as LiteLLM and Envoy AI Gateway cover this well, and the API-management vendors have followed. The architectural payoff is that model choice becomes configuration rather than a code change, which matters when applications live for years and models are superseded in months. The risk is the classic middleware one: an under-resourced gateway team becomes the bottleneck for every AI initiative in the company, so staff it as a product, not a proxy.
Enterprise GenAI Reference Architectures
A workable GenAI reference architecture is layered, and the layers change at different speeds. On top, the experience layer: copilots, chat surfaces, and APIs consumed by existing applications. Beneath it, orchestration: agent runtimes, prompt and workflow management, tool execution. Then the model layer, reached only through a gateway, and beside it retrieval: embedding pipelines, vector and hybrid search, document-level permissions. Underneath, the data layer that feeds retrieval and fine-tuning, which is where most programs actually stall. Cutting across everything, a governance plane: evaluation, guardrails, observability, cost attribution, and audit, delivered as shared services rather than rebuilt per use case. The architect's test for any such diagram is substitution: if you cannot swap the model, the vector store, or the orchestration framework without rewriting the experience layer, you have drawn a stack, not an architecture. That layering, not the specific products in the boxes, is what buys optionality as the market moves.
Integration Patterns: Connecting AI to Enterprise Systems
Connecting AI to enterprise systems is an integration problem before it is an AI problem, and the classic patterns apply with new failure modes. Synchronous request-response fits copilots and in-app assistance, but LLM latency is measured in seconds, not milliseconds, so streaming to the user, aggressive timeouts, and circuit breakers are mandatory. Asynchronous patterns fit document processing and agentic work better: requests land on a queue, workers call models, results publish as events, and the retry and dead-letter semantics you already run on Kafka or a service bus carry over directly. Event-driven agents invert the trigger: a business event (claim filed, invoice received) wakes an agent instead of a user prompt. None of this replaces the ESB or iPaaS; AI capabilities should surface there as just another API-first service with a contract, a version, and an SLA. The anti-pattern to police is point-to-point wiring, every team calling models directly from application code, which recreates the integration spaghetti the last decade spent removing.
Identity, Networking & Tenancy for AI Workloads
AI workloads inherit every identity and network requirement of ordinary services, plus one that is routinely missed: the model must never see data the requesting user is not entitled to. Workload identity comes first: agents and pipelines authenticate with managed identities or federated credentials, never long-lived API keys pasted into config. Entitlements then have to flow end to end: the user's token, or an on-behalf-of exchange, must reach the retrieval layer so document ACLs are enforced at query time, because filtering inside the prompt after retrieval is not a security boundary. On the network side, model endpoints belong behind private connectivity (Private Link, VPC endpoints) with controlled egress, since prompts are a convenient exfiltration channel. For multi-tenancy, tag every request with tenant and cost center at the gateway; token spend is the new cloud bill, and finance will ask who consumed it. Most GenAI incidents in practice are authorization gaps in retrieval, not model exploits.
Build vs Buy vs Orchestrate
The build-versus-buy question for AI splits into three tiers with three different answers. Buy the commodity: horizontal copilots for productivity suites, coding, and CRM, where a vendor amortizes R&D across millions of seats at something on the order of $30 per user per month, and you will never match the integration depth. Assemble the platform: gateways, vector stores, evaluation harnesses, and serving are selection and configuration work, not greenfield engineering. Build only where you differentiate: the domain agents and workflows that encode how your business actually operates, because no vendor ships your underwriting rules or your supply-chain heuristics. In practice the durable in-house asset is the orchestration layer: the code that binds models to your systems, entitlements, and processes, and survives every model swap beneath it. The recurring failure mode is inverted effort, teams rebuilding chat UIs and RAG plumbing they could buy while the genuinely differentiating workflow logic stays in slideware.