Multimodal AI: Beyond Text

The enterprise is not just text

Walk into any large organization and the information that runs it is overwhelmingly not clean text. It is scanned contracts and stamped invoices, engineering diagrams and CAD exports, product photographs, X-rays and dashboard screenshots, recorded sales calls and support voicemails, security footage and training videos. The first generation of enterprise assistants could touch almost none of it. They read the text you had already extracted and transcribed, which meant they operated on a thin, lossy projection of the real corpus and quietly inherited every error the extraction step made upstream.

Multimodal models move past that ceiling. A multimodal model processes, and sometimes generates, more than text: chiefly vision (images), audio (speech), and increasingly video. Instead of a text-only assistant sitting downstream of an OCR or transcription pipeline, you get a system that can look at the pixels of a page and hear the audio of a call directly. The difference is not cosmetic. A model that reads a document image sees the table borders, the checkbox that is ticked, the handwritten annotation in the margin and the logo in the corner, all of which a flattened text stream threw away.

This widens the addressable use-case surface sharply, and that is the strategic point for anyone deciding where to invest. Every process that was previously off-limits because its inputs were not text (visual inspection, claims triage from photos, meeting understanding, accessibility) becomes a candidate. The rest of this piece is about how that capability actually works, where it genuinely beats the old pipeline, and the costs that arrive with it, because the widened surface is real but so is the bill.

How a model sees

The mechanism is more concrete than the marketing implies, and understanding it tells you exactly where the capability is strong and where it is fragile. A vision-language model does not run optical character recognition and paste text into a prompt. It passes the image through a vision encoder, typically a vision transformer, which cuts the image into a grid of patches and turns each patch into a feature vector. A projection layer then maps those vision features into the same embedding space the model uses for text tokens. From the transformer's point of view, the image has become a sequence of vectors that live alongside the text tokens, and self-attention lets every text token attend to every image patch and vice versa.

  image ---> [ vision encoder ]  patches -> feature vectors
                     |
                     v
             [ projection layer ]  map into text embedding space
                     |
                     v
  "what is the total?"  ->  [ t1 t2 t3 ... ] text tokens
                     |
                     v
             [ transformer: self-attention over BOTH ]
                     |
                     v
                  answer, grounded in pixels + words
            

Contrast this with the classic alternative: a pipeline of dedicated components, OCR for documents or ASR for speech, whose text output is fed to a text-only model. That pipeline is modular and inspectable, but it commits to a single interpretation early. Once OCR decides a smudged digit is an 8, the language model never sees the pixels to reconsider, and layout, color, and spatial relationships are gone. A native multimodal model defers that commitment, letting the question ("what is the total on this invoice?") shape how it reads the image. That joint reasoning is the entire advantage, and it is also why native models can confidently misread fine detail: nothing forced an explicit, checkable transcription step in between.

The modalities, one by one

Multimodal is a bucket word, and the three modalities inside it are at very different levels of maturity. Treating them as one capability is the fastest way to over-promise. It helps to take them separately, because the enterprise value, the failure modes, and how much you should trust them today all differ.

Vision is the most mature and the most immediately useful. Modern vision-language models do genuine image understanding: they read charts and infer the trend, extract fields from a form without a trained OCR template, describe a photograph, and answer questions about a diagram. The headline is OCR-free extraction, pulling structured data straight from a document image, but the deeper value is that the model reads layout and content together.

Audio has moved past transcription. The interesting capability is speech understanding: summarizing a call, classifying sentiment and intent, flagging a compliance breach, answering questions about what was said, sometimes directly from the waveform rather than a transcript, so tone and hesitation survive. Video is the frontier: frames over time, which multiplies the token cost and the evaluation difficulty, and where claims deserve the most skepticism.

Where it earns its keep

Multimodal earns its budget in exactly the places where the old extract-then-reason pipeline loses information at the seam. The test is simple: if the meaning lives in the arrangement of the content and not just its text, a native model that reads the whole artifact tends to win. A few cases recur across enterprises, each of them a spot where a brittle pipeline stumbles.

• Document intelligence: reading invoices, contracts, and forms where the layout carries meaning. The model uses the position of a number, the column it sits under, and the checkbox beside it, so it survives the messy, non-standard documents that break template-based OCR. See the Document Intelligence lab.
• Visual inspection: spotting a defect in a product photo, damage in a claims image, or a wiring fault in a field snapshot, tasks that never had a text representation to feed a text-only model.
• Chart and diagram reading: answering questions about a bar chart or an architecture diagram. A pipeline would have to detect, digitize, and re-serialize the chart; the model reads the trend directly.
• Accessibility: generating rich alt text and describing visual content for screen readers, at a quality that rule-based tools never reached.
• Meeting and call understanding: going from a recording to decisions, action items, and risk flags, where tone and who-said-what matter as much as the words.

The common thread is that each of these has a pipeline alternative, and each pipeline alternative degrades precisely where the input is non-standard: the unusual invoice layout, the low-light inspection photo, the crosstalk on a call. Native multimodal does not eliminate those failures, but it moves the decision inside a single model that can weigh the whole artifact, rather than compounding errors across a chain of narrow components. That is where the capability stops being a demo and starts closing a process.

The hidden costs

The capability is genuine; so is the bill, and it arrives in four forms that are easy to underestimate from a slick demo. The first is tokens. An image is not free to attend to: the vision encoder turns it into many patch vectors, and to the model those behave like a long run of tokens. A single high-resolution page or photo can expand into hundreds or low thousands of tokens depending on resolution and how the model tiles it (numbers here are illustrative and vary widely by model). Feed a model a fifty-page document as images and you can spend more tokens on the pixels than on any text prompt around them, which drives both cost and latency up fast.

The second cost is reliability on fine detail. Because a native model never commits to an explicit transcription, it can hallucinate a small digit, a decimal point, or a serial number with complete fluency and total confidence. For a marketing summary that is harmless; for a payment amount or a dosage it is not, and it argues for a verification step on anything financially or legally load-bearing. The third cost is evaluation: grading whether a model correctly read a chart or understood a call is far harder than string-matching a text answer, and most teams lack a labeled multimodal test set (more in Evaluation Methods). The fourth is governance: images and audio carry faces, voices, and sensitive documents, pulling biometric and privacy considerations into scope that a text pipeline never raised. Extend your privacy and PII controls to pixels and waveforms.

Native model or classic pipeline

The architecture choice is not native-versus-legacy, and framing it that way leads teams to over-buy the flashy option. It is a genuine trade-off, and the honest answer for most enterprises is a hybrid. A native multimodal model wins when meaning is entangled with layout, when inputs are non-standard, or when the reasoning has to weigh visual and textual evidence together. A dedicated OCR or ASR pipeline feeding a text model wins when you need deterministic, auditable extraction of a known field at high volume and low cost, and when an explicit transcription is exactly the artifact you want to store and check.

The pragmatic pattern most mature teams land on uses each for what it is good at. Run cheap, deterministic OCR or ASR to produce a first-pass transcript and to handle the high-volume, standard cases, then route the hard, non-standard, or high-stakes items to a native multimodal model that reads the original pixels or audio and reconciles them against the transcript. You keep the auditable intermediate artifact where you need it, you keep the cost down on the easy majority, and you spend the expensive joint-reasoning capability only where it changes the answer. The decision is per-workflow, not per-platform, and it should be made with your own volume and accuracy numbers in front of you.

The architect view

For the architect who owns this decision, multimodal is not a model upgrade you switch on; it is a design dimension you have to reason about deliberately, the same way you reason about latency or cost. The capability is real and the widened use-case surface is worth pursuing, but the teams that get burned are the ones that treated pixels and audio as free text. Four disciplines separate a deployment that holds up from a demo that does not survive contact with production.

1. Treat modality as a deliberate choice. For each workflow, decide whether the input truly needs to be seen or heard, or whether a cheaper text extraction carries the meaning. Reach for multimodal where layout, imagery, or tone is load-bearing, not by default.
2. Budget the token cost up front. Model the per-document and per-minute token bill before rollout, downscale and crop aggressively, and set latency and cost expectations against real image sizes, not text-prompt intuition.
3. Evaluate on your own documents. Build a labeled set from your actual invoices, calls, and photos, and measure the model there. Public multimodal benchmarks tell you little about how it reads your particular forms and accents.
4. Extend governance to images and audio. Pull pixels and waveforms into the same PII, retention, and access-control regime as text, and account for the biometric and consent questions that faces and voices raise.

Do those four things and multimodal becomes what it should be: a way to bring the majority of enterprise information that was never text into reach, with the costs named and the guardrails in place. Skip them and you have bought an expensive, confident system that occasionally reads a payment amount wrong and no one can explain why. The technology has earned its place in the stack. What it has not earned is a pass on the engineering discipline you apply to everything else.