Algorithmic Bias

A computer does not need prejudice to treat people unequally. Bias can enter through historical data that records past discrimination, through samples that under-represent some groups, through proxies such as a postcode standing in for ethnicity, through the target the model is told to optimise, or through how staff use its output. The effects are often repeated at scale, turning a small skew into thousands of unfair decisions. Testing compares error rates and outcomes across groups, and fixes range from rebalancing data to changing thresholds or adding human review. Removing a protected attribute rarely cures the problem, because other fields can reveal it. Bias is a measurable pattern, whereas fairness is the standard against which the pattern is judged, and different fairness measures can conflict. Lenders, recruiters and insurers that skip group-level testing tend to learn about bias from a complaint or a regulator's letter.

A hiring tool is trained on ten years of past recruitment decisions at a firm that mostly hired men. It learns to rank CVs mentioning certain sports clubs and women's colleges lower. An audit comparing shortlist rates by gender shows a wide gap. The firm withdraws the tool, rebuilds the training set, and adds a recruiter review of every rejected shortlist.