Anatomy of an LLM API

What goes over the wire

Strip away the SDK, the client library and the friendly method names, and a call to a large language model is something almost disappointingly ordinary: an HTTPS POST to an endpoint, with a JSON body, answered by a JSON response. There is no special protocol, no bespoke transport. If you have ever called a REST API, you have already called an LLM; you just had a library hiding the envelope. Learning to read that envelope is the single highest-leverage thing you can do to reason about what these systems cost, how they behave, and why they sometimes disappoint.

The shape is worth committing to memory because everything downstream inherits from it. Cost is a function of the tokens in the request body and the tokens in the response, nothing more exotic than that. Latency is shaped by how many tokens the model must generate and whether you asked to receive them incrementally. Reliability is decided by how you handle the handful of ways the exchange can fail. None of these is a mystery once you can see the payload; all of them stay opaque as long as the SDK is a black box.

   CLIENT                                       PROVIDER
   ------                                       --------
   POST /v1/chat/completions
   Authorization: Bearer sk-...
   Content-Type: application/json
        |    { model, messages, params }
        | ------------------------------------->  |
        |                                          | (inference)
        |    { choices, finish_reason, usage }     |
        | <-------------------------------------  |
   200 OK  (or a stream of deltas)

This deep-dive walks the anatomy in the order a request lives it: what you send, what comes back, how streaming reshapes the return trip, how the exchange fails and how you make it survive, and finally why so many providers agreed to speak the same dialect. The goal is not to memorize one vendor's field names but to hold the underlying structure, which barely changes across providers.

The request

The request body is a JSON object, and three parts carry the weight. First, model: a string naming which model should answer, the choice that sets both capability and price. Second, messages: an ordered array of turns, each with a role and content. The roles are conventional and few. A system message sets standing instructions and persona; user messages carry the human input; assistant messages hold the model's prior replies, replayed back so the model can see the conversation it is continuing. The model is stateless between calls, so this array is the entire memory it has.

Third, a set of generation parameters that tune how the model decodes. These are optional and have defaults, but they are where behavior is shaped, so they reward understanding rather than copy-paste. Two more optional fields matter for enterprise use: a tools (or functions) array of JSON schemas the model may call, covered in tool calling, and a response_format that constrains output to strict JSON, covered in structured outputs.

The important intuition is that the request is not a command so much as a context. You are assembling the entire world the model will reason within, then handing over the decoding knobs. Everything the model knows about your intent lives in these fields, which is why prompt work and parameter tuning are the same craft seen from two angles.

The response

The response is a JSON object, and three parts of it deserve your attention long before the generated text does. The first is choices, an array holding the completion, typically one entry whose message carries the model's reply (either ordinary content or a request to call a tool). The second and third, easy to skip past and expensive to ignore, are finish_reason and usage. Reading these two on every call is the difference between an integration that behaves in production and one that surprises you there.

finish_reason tells you why generation stopped, and each value points to a different action on your side:

• stop: the model finished naturally or hit a stop sequence. The answer is complete; take it as given.
• length: generation hit the max_tokens ceiling and the reply is truncated mid-thought. Treating this as a clean answer is a common and quiet bug; you either raise the ceiling or continue the completion.
• tool_calls: the model is not answering but asking to invoke a tool you offered. Your code runs the tool and feeds the result back as another message.

The usage block reports token counts, usually prompt_tokens for the input, completion_tokens for the output, and their total. This is not telemetry trivia; it is your invoice, itemized per call. Because input and output tokens are frequently priced differently, and output is often the pricier side, the split is what lets you attribute spend and spot waste before it compounds. A single request costs a fraction of a cent (figures here are illustrative and move often), but multiplied across a feature's traffic it is the whole of your inference bill.

Capturing usage on every call, tagged by team, feature and use case, is the raw material of any serious cost practice; it is what turns an undifferentiated provider invoice into unit economics you can actually manage. Metered at a gateway, it becomes the visibility layer the whole FinOps cycle depends on.

Streaming

By default the exchange is atomic: you wait while the model generates the entire reply, then receive it in one response. For a long answer that wait can run into seconds, and a user staring at a spinner feels every one of them. Streaming addresses this by changing the return trip. Instead of one JSON object at the end, the server holds the connection open and pushes a sequence of small events as tokens are produced, using server-sent events (SSE). Each event carries a delta, a fragment of the growing message, and the client stitches the fragments together as they arrive.

   POST /v1/chat/completions   { ..., "stream": true }

   data: {"choices":[{"delta":{"role":"assistant"}}]}
   data: {"choices":[{"delta":{"content":"The"}}]}
   data: {"choices":[{"delta":{"content":" answer"}}]}
   data: {"choices":[{"delta":{"content":" is"}}]}
   data: {"choices":[{"delta":{"content":" 42."}}]}
   data: {"choices":[{"finish_reason":"stop"}]}
   data: [DONE]

What streaming buys is perceived latency, not real latency. The total time to the last token is roughly the same; the win is that the first token appears fast, so the interface feels alive while the rest fills in. For any human-facing chat surface that trade is almost always worth making. For a machine consumer that needs the whole answer before it can act, such as parsing a JSON tool call, streaming adds complexity for a benefit no one experiences.

The cost lands in error handling and buffering. In the atomic model a failure is clean: the request either returns or it does not. In the streaming model the connection can drop after some tokens have already been delivered and rendered, leaving you holding a partial answer and needing a policy for it. You cannot inspect a status code up front and be done; you must handle mid-stream faults, reassemble deltas in order, and decide whether a truncated stream is retried, discarded or surfaced. Streaming is the right default for responsiveness, but it moves work onto the client, and that work is where teams underestimate it.

Errors, limits and retries

An LLM endpoint fails in the same ordinary ways any networked service does, and treating those failures as an afterthought is what separates a demo from something that survives production. The response you never want to skip planning for is HTTP 429, the rate limit. Providers cap how many requests and tokens you may send per minute, and under load, or during a traffic spike, you will hit that ceiling. A 429 is not an error in your request; it is a signal to slow down and try again shortly.

The correct response to a 429, or to a transient 500 or a timeout, is a retry with exponential backoff: wait a short interval, then a longer one, then longer still, ideally with a little random jitter so a fleet of clients does not retry in a synchronized wave that recreates the overload. Retries need a ceiling, both in count and in total time, so a struggling dependency degrades gracefully instead of amplifying the pressure. Naive retry loops with no backoff are a reliable way to turn a brief provider hiccup into a self-inflicted outage.

Two further concerns round out reliability. Timeouts must be set deliberately: generation genuinely takes time, so a timeout tuned for a fast microservice will sever healthy long completions, while no timeout at all lets a stuck request hang a worker indefinitely. And idempotency matters because a retry after an ambiguous failure can mean the model runs twice; where a duplicate would double-charge or double-act, an idempotency key lets the provider recognize the repeat and return the original result rather than doing the work again. These are unglamorous mechanics, and they are precisely the ones that decide whether the integration holds up when real traffic arrives.

The de-facto standard

There is a quietly remarkable fact underneath everything above: much of it looks the same no matter whose model you call. The request shape, model, a messages array of roles, the familiar generation parameters, and the response shape, choices with a finish_reason and a usage block, first popularized by one provider's chat completions endpoint, have been widely adopted as a de-facto standard. A large share of providers, and nearly every gateway and open-source serving stack, now expose an OpenAI-compatible interface, whatever model actually runs behind it.

This was not mandated; it settled. The ecosystem of client libraries, tutorials and tooling built around that shape grew large enough that speaking a different dialect became a self-imposed tax. New providers chose compatibility because it let them inherit an existing world of clients on day one, and the standard reinforced itself with each adoption. Standardization here is not elegance for its own sake. It is the practical foundation of portability.

The payoff is that models become interchangeable behind a stable interface. Because the envelope is shared, moving a workload from one provider to another, or splitting traffic across several, is closer to a configuration change than a rewrite. The differences that remain, tokenizer quirks, exact parameter ranges, which capabilities a given model supports, are real and worth testing, but they are seams to smooth over rather than a wall to rebuild against.

The architect view

The reason to learn the wire format is not academic completeness; it is that the three questions a CTO actually asks about an AI system all resolve to the anatomy. What does it cost? Read the usage block: cost is tokens in plus tokens out, and nothing you can optimize lives outside that arithmetic. Why is it slow? Look at how many tokens it must generate and whether you stream them, because time-to-first-token and time-to-last-token are different problems with different fixes. Will it hold up? Trace how the exchange fails, whether you back off on 429, whether a retry can double-charge. Each answer is a field on the wire, not a vibe about the model.

This is why the anatomy is load-bearing knowledge for anyone designing on top of these systems rather than merely consuming them. A team that cannot read the payload will treat cost as a surprise, latency as luck, and reliability as hope. A team that can read it treats all three as engineering, measurable, attributable, and improvable. The wire format is the shared language in which those conversations become concrete.

Hold that frame and the rest of the estate falls into place. Cost discipline is meter the tokens and manage unit economics. Latency work is stream where a human waits and cap generation where they do not. Reliability is back off, bound retries, and keep them idempotent. And portability is a gift of the standard the whole industry happened to agree on. You can feel the shape of a request by hand in the Prompt Lab, but the anatomy you build against is the same everywhere the wire runs.