Model Validation

Validators look at three broad areas. First, the design: whether the theory, assumptions and data suit the use. Second, the implementation: whether the code or system does what the documentation says. Third, the results: whether outputs hold up when compared with actual outcomes and with alternative approaches. Typical tests include back-testing, sensitivity analysis, benchmarking against another model and review of data quality. Validation takes place before a model is first used, after material changes, and at regular intervals. Findings are rated, and weak models are limited, fixed or withdrawn. Independence is the key condition: reviewers must be able to challenge developers without pressure from the business line that wants the model approved. Without it, a model goes live on its developer's word alone, with nobody placed to say where it breaks.

A card lender's team builds a new loss forecasting model. The validation unit reruns the code, tests the model on a data set it has never seen, and finds that it underestimates losses for new customers. It rates the model as usable only with an added adjustment and sets a six-month review. The business accepts the limit before launch.