Explainability

Some models, like a simple scorecard, can be read directly. Others, such as deep neural networks, are effectively black boxes, so analysts add tools that approximate their reasoning, for example by showing which inputs weighed most on a given result. Explanations come at two levels: global, describing how the model behaves overall, and local, explaining one decision for one person. Different audiences need different detail. A customer wants the main reasons for a refusal, a validator wants to see that the logic is sound, and a regulator wants evidence that no prohibited factor played a part. There is often a trade-off between accuracy and ease of explanation. A model that cannot give reasons for a refusal is hard to defend in a complaint or an appeal, and validators often hold it back for that reason.

An energy supplier's model rejects a customer's application for a payment plan, and the customer asks why. The system lists the three factors that pulled the score down most: two missed bills, a high balance and a short account history. The agent checks these against the file, confirms they are accurate, and writes back explaining how to reapply once the position improves.