Considering intrinsic versus post hoc explainability
Can we ignore explainability and just trust state-of-the-art models? The short answer is no. A correct prediction does not adequately solve real-world problems. Users will not adopt an AI system unless it is trustworthy. Knowing why predictions are made is important to show how much an AI system can be trusted and provide insights into the best course of action when combined with domain expertise.
There are no hard-and-fast rules when choosing intrinsic versus post hoc explainability. For explainability’s sake, an intrinsic explainable model with adequate accuracy is always preferable over a complex model. Otherwise, post-processing explanation methods are alternatives to provide post hoc explainability after model training.
Inevitably, humans trust explanations selectively with unconscious bias based on profession, domain knowledge, and personal experience. Often, humans seek contrastive and interactive explanations...