Backtesting

A model is only worth trusting if its forecasts hold up. Backtesting takes past predictions, such as the daily loss a Value at Risk model said would be exceeded one day in a hundred, and checks them against real results. Too many exceptions suggest the model understates risk. Far too few may mean it is overly cautious and ties up capital. Banks run it routinely on market risk and credit models, and Basel rules count exceptions to set capital add-ons. The same idea applies elsewhere, for example testing a fraud score against confirmed fraud cases. Backtesting looks backward, so it cannot prove a model will cope with conditions that have not yet occurred. Validators and supervisors read its exception counts when deciding whether a model stays in use, gets recalibrated or draws a capital add-on.

An insurer's investment team uses a one-day 99% VaR model. Over 250 trading days, losses exceeded the VaR figure on nine days, where about two or three would be expected. Model validation concludes the model understates risk and reports this to the risk committee, which orders a recalibration.