Pillar 3 - Explainable Outputs

The Question

Can you defend it?

Coverage is what you see. Timing is when you see it. Defensibility is what happens when somebody asks why. A CFO who can’t answer that question can’t sign the certification. An auditor who can’t answer that question can’t issue the opinion. A regulator who can’t answer that question can’t close the examination. The whole point of oversight is that its outputs survive the question. A platform that produces findings nobody can defend has produced nothing of governance value, no matter how sophisticated the technology underneath.

This pillar is the defensibility argument.

The Defensibility Test

There are three audiences who routinely ask “why did this get flagged?” The platform’s outputs have to survive each one.

The internal auditor or controller. Investigates the finding before action. This audience needs to understand what specifically about the transaction triggered the flag, whether the pattern matches a recognized risk, and what evidence supports the conclusion. A score alone fails this test. An output of “Risk score: 0.87” doesn’t tell the controller what to look at or what to do next. The controller has to reconstruct the reasoning from scratch, which means the platform’s output wasn’t doing its job.

The external auditor or regulator. Reviews the finding as part of an audit or examination. Here, these validators need to understand the methodology, evaluate whether the methodology is sound, and verify that the conclusion follows from the inputs. The standard isn’t “the platform got the right answer.” The standard is “we can independently reconstruct the platform’s reasoning and agree with it.” This is the test SR 11-7 was written to formalize, and the test the EU AI Act now imposes by binding regulation.

The board or audit committee. Reviews aggregate findings, asks about specific cases, and ultimately accepts or rejects the certification. This group needs to understand what the platform flagged, why it matters, and what the organization did about it. The board does not have the time or technical background to interpret raw scores. They need the platform’s outputs translated into business-meaningful language that a non-technical executive can defend in front of investors and analysts.

A finding that survives all three tests is defensible. A finding that survives only one is not.

Why Black-Box AI Fails

The financial AI conversation in recent years was dominated by large language models. The current conversation is increasingly about agentic workflows: AI systems that execute complex financial tasks across enterprise processes.

These systems are powerful. But as oversight outputs in their own right, they fail the defensibility test by architecture.

The first problem is reproducibility. LLMs are probabilistic systems. The same prompt against the same model can produce different outputs across different runs. The variation may be small, but it is structurally present. An auditor reviewing a finding from last quarter cannot necessarily reproduce the exact output today. For governance purposes, “similar” is not the same as “the same answer.” Oversight requires outputs that can be reproduced and examined over time.

The second problem is inspectability. When an auditor asks why a transaction was flagged, the platform must be able to show the reasoning behind the finding. LLMs can generate explanations, but those explanations are not the same thing as transparent reasoning. The underlying computation spans millions of parameters and cannot be reconstructed in a way that allows an auditor or regulator to independently verify the path that produced the conclusion.

None of this is a criticism of LLMs as a class of technology. LLMs are valuable in many roles within an AFO platform: natural-language interfaces, narrative generation, conversational investigation, and workflow orchestration. The question is not whether LLMs are useful. The question is whether probabilistic outputs can satisfy governance requirements that demand reproducibility, attribution, and examination.

The relationship between agentic workflows and AFO is therefore better-together, not either-or. Agentic workflows built on LLMs are increasingly responsible for operational work across finance: closing the books, reconciling accounts, drafting reports, and investigating exceptions. Those workflows depend on authoritative inputs. An agent investigating a flagged variance is only as credible as the data and findings it receives.

The deterministic outputs of an AFO platform, reproducible findings, attributable reasoning, and traceable lineage, provide that foundation. Once the findings are defensible, agents can amplify their reach without diluting their authority.

AFO provides the oversight layer that allows agentic workflows to operate credibly in regulated environments.

What Deterministic Means

Deterministic, in the AFO context, means a specific architectural property: the same inputs produce the same outputs, every time.

If the platform processes the same transaction population on Monday and again on Friday with no data changes in between, it produces the same findings, with the same risk scores, attributed to the same features, supported by the same reasoning. The output is reproducible. An auditor can pull last quarter’s findings, rerun the analysis, and get the same result. The platform’s reasoning is not a snapshot of a stochastic process; it is a function of the inputs.

Deterministic outputs require the underlying detection engine to be built on architectures that produce repeatable results. Statistical models, business rules, and unsupervised machine learning trained on bounded historical data can all be deterministic if the platform commits to the property. Probabilistic LLMs, by contrast, are deterministic only if explicitly constrained (low temperature, fixed seeds, controlled context), and even then the determinism is environmental, not architectural.

The architectural commitment matters because audit and regulatory frameworks require reproducibility, not approximation. The SEC’s 2026 examination priorities explicitly review whether AI tools can demonstrate the logic behind their flagged decisions. The standard across these frameworks is the same: the platform’s reasoning is examinable, and the examination produces the same answer the platform produced.

Feature Attribution Beats Risk Scores

A risk score is not an explanation. It is a summary number that compresses a finding into a single dimension. By itself, it tells the recipient that something is risky but not why or what to do.

Feature attribution is what completes the score. For each finding, the platform identifies the specific transactional features that produced the elevated risk: the timing pattern, the vendor relationship, the amount distribution, the approval anomaly, the missing reference document. The score says “this is risky.” The attribution says “because of this combination of factors.”

The difference matters operationally. A controller receiving a risk score of 0.87 has no information about what to investigate. A controller receiving the same score with the attribution “three payments of $9,999 within 48 hours to a new vendor, each just below the $10,000 approval threshold” knows immediately what to look at and what story to construct. The attribution turns the alert into something actionable.

The difference matters defensibly. An auditor reviewing a flagged transaction can verify the attribution against the underlying data. The vendor was new; the platform’s data shows the vendor master record. The amounts were just below threshold; the platform’s data shows the amounts. The pattern was unusual; the platform’s data shows the historical baseline. Each element of the explanation is grounded in transactional evidence the auditor can inspect independently. The finding becomes a structured argument the auditor can evaluate, not an opaque conclusion they have to accept.

The Regulatory Convergence

Four governance frameworks converge on explainability as a binding requirement.

EU AI Act Article 13 requires high-risk AI systems to be sufficiently transparent for deployers to interpret outputs. The article specifies mandatory documentation of accuracy, robustness, foreseeable misuse, logging mechanisms, and intended purpose. Financial AI deployed in credit, fraud detection, and decisioning workflows falls within high-risk scope. Compliance is not optional for systems deployed in the EU after August 2, 2026.

EU AI Act Article 14 requires effective human oversight, including the ability to “correctly interpret outputs.” The article presumes the outputs are interpretable in the first place. A platform whose outputs cannot be interpreted by the human overseer fails the obligation by construction.

Federal Reserve SR 11-7 has required documentation sufficient for unfamiliar parties to understand model operation since 2011. The standard pre-dates the AI explainability conversation by a decade. Bank regulators are now applying SR 11-7 to AI models in production, which means the explainability bar is whatever the original supervisory guidance required: a third party w