ModelRefs / Explainable AI (XAI) — AI Glossary
Explainable AI (XAI) — AI Glossary
The field developing techniques that make AI model decisions understandable to humans, supporting accountability and trust.
Overview
XAI encompasses: post-hoc explanation (LIME, SHAP), inherently interpretable models (decision trees, linear models), example-based explanations (counterfactuals, influential training examples), and concept-based explanations (TCAV). DARPA XAI program (2017–2021) catalyzed the field. EU AI Act mandates explanations for high-risk AI decisions.
Reference details
| Topic | safety |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Example: The explanation explains the surrogate
A post-hoc method reports which input features mattered by probing the model from outside — perturbing inputs and watching outputs, then fitting something simple to that local behaviour. What you get is a faithful account of the simple thing, and an approximation of the model. Two methods run on one prediction routinely disagree, and one method run twice can disagree with itself when its sampling is random. That is not a reason to abandon them; it is a reason to state what they are. An explanation is evidence about behaviour near one input, not a description of the model's reasoning.
Commonly confused with
Explainability and interpretability are used interchangeably but pull apart usefully: interpretability is a property of a model you can read directly, explainability is the practice of accounting for a model you cannot. A chain-of-thought trace is neither — it is generated text about the answer, not a record of the computation that produced it, and it can be fluent and wrong.
When to use it
Reach for it when:
- Regulated decisions where a rationale must be produced and defended
- Debugging: finding which feature or token a model is keying on
- Auditing for proxy discrimination, where the question is what stood in for a protected attribute
Reach for something else when:
- As proof of causation, or as a guarantee the model will behave that way elsewhere
- Treating a self-reported rationale as an explanation of the computation
- Where an inherently interpretable model would do the job — then you need no surrogate
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Explainable AI (XAI) — AI Glossary.
Frequently asked questions
What is Explainable AI (XAI)?
The field developing techniques that make AI model decisions understandable to humans, supporting accountability and trust.
What concepts are related to Explainable AI (XAI)?
Closely related concepts include interpretability, lime, shap.