Design Effectiveness and Operating Effectiveness

A control can fail in two separate ways. It can be badly designed, for example a review by someone who lacks the information or authority to catch errors. Or it can be well designed but not carried out, for example reviews that are skipped in busy months. Auditors test both. Design is usually assessed first, by walking through the process, reading procedures and talking to those who perform the control. Operating effectiveness is then tested over time by sampling instances and looking for evidence that the control ran, on schedule, by the right person, with exceptions followed up. A poorly designed control cannot be operating effectively in any useful sense, so there is little point testing it further. The pair is central to management's own assessments under rules such as the Sarbanes-Oxley Act. Reports that cover only one of the two tests can leave readers falsely reassured.

A retailer requires a manager to review a weekly list of refunds. Auditors read the procedure and find it sound, so the design is effective. They then sample twenty weeks and find no sign-off in seven of them. The control is rated as not operating effectively, and the retailer must fix how it runs, not how it is written.