Explainability and Interpretability: What Can You Actually Know?

The question everyone asks

Every enterprise AI system eventually faces the same question: why did the model say that? It arrives from an auditor reviewing a flagged transaction, from a regulator examining a lending decision, from a customer disputing an outcome, or from an engineer staring at a bad output in production. The words are identical. The answer each person actually needs is not.

Treating "explainability" as one deliverable is the first mistake most programs make. In practice there are at least four distinct demands hiding behind the question:

• Auditors and regulators need a defensible account: what inputs were considered, what policy governed the decision, and evidence that the process was followed. They care about process integrity more than neural activations.
• Affected customers need actionable reasons: what drove the outcome and what, if anything, they could change. Vague references to "the algorithm" satisfy no one and, in some regulated domains, may not satisfy the law either.
• Engineers need causal debugging information: which prompt version, which retrieved documents, which tool results shaped this specific output, so they can reproduce and fix the failure.
• Executives and risk owners need a bounded claim: what the system can and cannot be relied upon to do, stated honestly enough to sign off on.

A single artifact cannot serve all four audiences, and pretending it can produces explanations that are simultaneously too technical for customers and too shallow for auditors. The rest of this article draws the line between what research can tell you about a model's internals, what you can engineer around the model today, and where the honest answer is still "we do not fully know."

Interpretability vs explainability

The two words are used interchangeably in vendor decks, and they should not be. Interpretability is about understanding the internal computation: which components of the model do what, and how the parameters transform an input into an output. Explainability is about producing a usable account of why a particular output happened, at a level of abstraction the audience can act on. One is a property you investigate; the other is a deliverable you construct.

Mechanistic interpretability, the research program that tries to reverse-engineer what circuits inside a network are computing, is real science with genuine progress: researchers have identified interpretable features and circuits in large models. But it is early. As of mid-2026 it does not give you a faithful, complete account of why a frontier LLM produced a specific answer in production, and no honest vendor will tell you otherwise. Explainability, by contrast, is achievable today, because it is a systems-engineering problem rather than a neuroscience problem.

The conflation matters because procurement and governance conversations regularly demand "interpretability" when what the organization needs, and can actually get, is explainability. Specify which one you mean before you write either into a policy.

Why LLMs resist explanation

A large language model is billions of parameters applied in parallel, with concepts stored as distributed representations: no single weight holds "creditworthiness" or "sarcasm," and any given neuron participates in many unrelated behaviors. There is no decision path to print, no rule that fired. The computation that produced a specific answer is spread across the entire network in a form that current tools can only partially illuminate.

Faced with that opacity, teams reach for the obvious shortcut: ask the model to explain itself. It will happily comply, and that is exactly the trap. A self-explanation is generated by the same next-token process as every other output. The model is producing plausible text about a decision, not reading out the computation that made it. The explanation can cite factors the model never used, omit ones that mattered, and shift between runs. Reasoning traces and chain-of-thought output are useful engineering signals, worth logging and inspecting, but they carry the same caveat: they are text the model produced, not verified ground truth about its internals.

This is not a temporary tooling gap that the next model release fixes. It follows from what these systems are. Plan your governance posture around it rather than waiting for it to go away.

What works for classic ML

The picture is meaningfully better on the predictive-ML side of the house, which is one more reason the predictive ML vs GenAI distinction matters for governance. For tabular models such as gradient-boosted trees, feature-attribution methods in the SHAP and LIME family give workable answers: for this applicant, income contributed this much toward approval, utilization this much against. The attributions are approximations with known caveats, and they explain the model's behavior rather than the world, but they are stable enough to support review processes and reason codes, and practitioners have years of accumulated experience with their failure modes.

Better still are models that are interpretable by construction: linear and logistic models, scorecards, small decision trees, and constrained additive models. You do not approximate their reasoning after the fact; you read it directly off the coefficients or the tree. Where explanation is mandatory, inherently interpretable models remain the gold standard, and in many tabular problems the accuracy sacrificed to get that property is smaller than teams assume. The right default in high-stakes deciding roles is to try the interpretable model first and force the complex model to justify its marginal lift.

What does not work is transplanting this toolkit to LLMs. Feature attribution assumes a modest, fixed set of input features; an LLM's input is thousands of interacting tokens, and its "features" are internal representations nobody enumerated. Token-level attribution methods exist and can be directionally useful for debugging, but they do not produce the kind of stable, defensible account that a reason code provides. The tools did not fail; the object changed underneath them.

Traceability: the explainability you can build

If you cannot explain the weights, explain the system. For LLM applications, the explainability an enterprise can actually deliver is traceability: a complete, replayable record of everything that surrounded the model call. What context did the model see? Which documents did retrieval supply, from which source and version? Which tools were called, with what arguments and results? Which prompt template and model version were live? Who saw the output and what did they do with it? None of this requires solving interpretability, and all of it is buildable with the same infrastructure described in the tracing and observability deep-dive.

DECISION TRACE  req_8f31          2026-07-02T14:07:55Z
  prompt     claims-triage v14    model llm-large (2026-06)
  input      "summarize claim #58821 and flag exclusions"
  retrieval  policy-kb: DOC-112 s3, DOC-207 s1  (top 2 of 8)
  tool call  claims_api.get_status(58821) -> "under review"
  output     summary + 2 citations [DOC-112 s3] [DOC-207 s1]
  checks     citation-valid PASS   pii-scan PASS
  review     human opens DOC-112 s3 and verifies the claim
            

The second pillar is verifiable citations. When the system grounds its answers in retrieved sources and cites them at claim level, a human reviewer does not need to trust the model's reasoning at all: they check the citation against the source, as covered in the grounding and citations deep-dive. That shifts the explainability question from "why did the model think this?" to "does the cited evidence support this?", which is a question people can actually answer.

This is a weaker claim than knowing the model's internals, and you should say so plainly. But for most audit, debugging, and accountability purposes, a faithful account of what the system did beats a speculative account of what the network computed.

When the law expects an answer

In some domains, explanation is not a nice-to-have but an expectation with teeth. US credit decisioning is the canonical example: adverse-action requirements have long obliged lenders to give applicants specific, accurate reasons when credit is denied. Similar reason-giving expectations appear, in varying forms, around employment, housing, and insurance decisions, and US regulators have signaled that using an algorithm does not waive the obligation. The details vary by statute and jurisdiction and continue to evolve, so treat this section as orientation and route specifics through counsel. The direction of travel, though, is consistent: automated decisions in these domains must come with reasons a specific person can understand and contest.

Now put that requirement next to the previous two sections. An LLM cannot give you a faithful account of why it decided, and its self-explanations are inadmissible as ground truth by your own engineering standards. The architectural conclusion writes itself: where the law expects specific reasons, keep an inherently interpretable model in the deciding seat, one whose reason codes are real, and scope the LLM to assistive roles around it. Let the LLM summarize files, draft correspondence, answer applicant questions, and restate the interpretable model's actual reasons in plain language. The decision, and the reasons for it, come from the component that can truthfully provide them.

The architect view

Explainability is not one control; it is a set of per-use-case decisions about what kind of account is owed, to whom, and by which component. The NIST AI Risk Management Framework lists explainability and interpretability among the characteristics of trustworthy AI, which makes this a named line item in most governance reviews; your job is to turn that line item into architecture. In practice that means three moves.

1. Classify the explanation debt per use case. For each system, write down who can demand an account (regulator, customer, auditor, engineer), what form it must take, and what the law expects. A marketing-copy assistant and a credit model do not carry the same debt and should not carry the same machinery.
2. Build traceability as a platform property, not a per-team habit. Traces, prompt and model versioning, retrieval provenance, and citation checking belong in the shared platform layer so every application inherits them by default. Explainability retrofitted after an incident is expensive and unconvincing.
3. Match the model class to the explanation requirement. Interpretable models where reasons are legally owed; classic ML with feature attribution where approximate accounts suffice; LLMs where traceability and verifiable grounding meet the need. The requirement picks the model, not the other way around.

Then comes the part that earns trust with skeptical stakeholders: be honest about the frontier. Say plainly that nobody can yet fully explain why a frontier LLM produced a specific output, that self-explanations are narratives, and that your controls are designed around that fact rather than in denial of it. In a market saturated with confident claims, the architect who states what cannot be known, and shows the traceability that answers what can, is the one the auditors, and the board, end up believing. Try the failure modes yourself in the Governance Sandbox, or continue with the rest of Governance.