Professional working-party output on explainable AI has done useful myth-busting on the "black box" framing, and supervisors have signalled that AI in regulated decisions is evaluated by domain and consequence rather than by architecture. This article draws the practical conclusion for validation.
Explainable AI model validation in insurance is usually discussed the wrong way round. Explainability gets framed as a product feature - something bolted on for customer communications or regulatory comfort. The stronger position, and the one implicit in how regulators actually examine models, is that explainability is a validation requirement: a model whose behaviour cannot be interrogated cannot be validated in the sense that validation standards mean, whatever its holdout accuracy.
Validation asks whether a model is conceptually sound, whether it behaves appropriately across the space it will meet in production, and whether its ongoing use remains justified. Every one of those judgements requires visibility into why the model does what it does. Accuracy metrics answer "how often is it right on data like the past." Only explanation answers "is it right for reasons that will persist, and wrong in ways we can anticipate."
Retiring the black-box excuse
The claim that modern machine learning models are inherently inexplicable has not survived contact with the tooling. Global structure can be interrogated with permutation importance and partial dependence; local decisions with Shapley-value attributions; behaviour with counterfactual probes - what minimal change flips this decision; and whole models with distilled surrogates that reveal the dominant logic. Professional working-party output has made this point firmly: the black box is, for most tabular insurance applications, a choice of effort level rather than a fact of nature.
Two disciplines keep the tooling honest. First, faithfulness: an explanation must describe what the model actually computes, not a plausible story beside it - surrogate models in particular need their fidelity measured and reported. Second, stability: attributions that swing wildly for near-identical cases are telling you about the model, and what they are telling you is itself validation evidence.
What explanation-based validation catches that accuracy cannot
Leakage and spurious features. The classic catch: a model performing brilliantly because a feature encodes the outcome. Attribution analysis surfaces it in an afternoon; accuracy testing never will, because the leak improves accuracy.
Fragile logic. Partial-dependence shapes that reverse direction implausibly, or interactions that exist only in a sparse corner of the data, mark behaviour that will not survive distribution shift.
Proxy discrimination. Attribution across groups reveals when a facially neutral feature is doing the work of a protected one - the analysis fairness review needs and aggregate metrics obscure. In pricing contexts this connects directly to the territorial and climate-fairness questions I have written about elsewhere: the explanation layer is where "accurate" and "defensible" are forced to meet.
Silent behavioural drift. Monitoring attribution distributions in production catches a model whose accuracy holds while its reasons shift - the precursor of failure under regime change, invisible to performance dashboards.
There is one honest caveat, which readers of my piece on LLM actuarial reasoning will recognise: fluent explanation can outrun competence. An explanation layer that generates persuasive narratives is a risk of its own if faithfulness is not measured. The requirement is explanations that are true, not explanations that are good.
Calibrating depth to consequence
Supervisory logic evaluates models by the decisions they inform, not the architecture inside them - a lending decision attracts lending-grade scrutiny whether it comes from a scorecard or a gradient boosting machine. Explainability requirements should scale the same way. A claims-triage assistant needs enough interrogability for its owner to detect misbehaviour. A pricing model in a regulated personal line needs feature-level accountability sufficient to survive a rate-filing challenge and a discrimination review. An automated decision affecting an individual's access to cover needs a case-level explanation a customer and an ombudsman can understand - which UK consumer-protection expectations effectively require regardless of modelling choices.
This gives validators a clean question set for any AI system: who needs to understand this model, to what depth, to discharge which obligation? The answers - owner, validator, regulator, customer - define the explanation artefacts the system must produce. Build them into the model's acceptance criteria, and "explainability" stops being an aspiration and becomes a deliverable.
The practical standard
A model is validation-ready when: its material drivers are known and defensible; its local decisions can be attributed faithfully; its behaviour under counterfactual and shifted inputs has been probed; its explanations are stable, measured for fidelity, and monitored in production; and the depth of all of the above matches the consequence of the decisions it informs. That standard is demanding, achievable with current tools, and - increasingly - simply what competent validation of consequential models means.
The firms treating explainability as a compliance garnish are building models they cannot defend. The firms treating it as validation evidence are building models they actually understand. The second group will have better models, not just better paperwork - because the process of explaining a model is, reliably, the process of finding what is wrong with it.
Key Takeaways
- Explainability is validation evidence, not a product feature: conceptual soundness and appropriateness cannot be judged without visibility into model reasoning.
- The black-box framing is largely retired for tabular insurance applications - attribution, partial dependence, counterfactuals and surrogates make interrogation an effort choice.
- Explanation-based validation catches what accuracy cannot: leakage, fragile logic, proxy discrimination and silent drift in model reasoning.
- Explanations must be faithful and stable - measured properties, not narratives; persuasive-but-unfaithful explanation is itself a risk.
- Depth of required explainability scales with decision consequence and audience - owner, validator, regulator, customer - and belongs in acceptance criteria.
Frequently Asked Questions
Why is explainability required for model validation? Because validation judges whether a model is right for reasons that will persist - conceptual soundness, appropriate behaviour, justified ongoing use. Accuracy metrics only measure past performance; explanation is the evidence that lets a validator anticipate failure modes, detect leakage and assess fairness.
What explainability techniques work for insurance models? For tabular models: permutation importance and partial dependence for global structure, Shapley-based attribution for individual decisions, counterfactual probes for decision boundaries, and measured-fidelity surrogate models for overall logic - with attribution stability and faithfulness reported as validation metrics.
How much explainability do regulators expect for AI models? Proportionate to the decision, not the architecture: regulated pricing and underwriting decisions attract feature-level accountability and, where individuals are affected, case-level explanations accessible to customers. The practical test is whether each audience - owner, validator, supervisor, customer - can discharge its obligation with the explanations provided.