Your AI Business Case’s Blind Spot
I have worked on hundreds of business cases. Every one was written inside a department or a division, and written for the benefit of that department or division. That is not a criticism. It is what a business case is for. The only one I ever saw that mentioned disaster recovery was a business case about disaster recovery. That was fine for thirty years. It is not fine now, and the reason has nothing to do with the quality of the cases or the people writing them. The AI business case and the continuity plan never meet Two owners, one assumption, no shared document An AI business case is built by a business unit that wants to be better at the thing it does. Faster lease abstraction, cleaner forecasts, fewer touches per invoice. The case compares the world with the project against the world without it, over three or five years, and both of those worlds are assumed to keep operating. Continuity belongs to somebody else. IT owns it, writes the plan, tests it on its own cycle, and reports on it to a different committee. The continuity plans I have read all have a section saying the work will be done manually and reconciled when service returns. Underneath that section sits an assumption: somewhere in the business are people who know how to do the work without the system. Not named. Not budgeted. Not tested. Present. That assumption survived mainframes, client-server, ERP, the web, mobile and the cloud. Every one of those waves changed how the work was done and left behind the people who knew how to do it. The process moved. The knowledge stayed in the building. The gap is not a missing number. It is a seam between two owners. And a second seam opens after the deployment goes live. The business has reduced its human fallback and increased its dependence on a provider it does not control. Two dependencies moving in opposite directions, neither of them recorded in the document that authorized the move. Nobody skipped a step. No step exists. Stated as a CFO would state it: the case captures the labor avoided. It does not capture the replacement cost of the capability being consumed, because that capability has never appeared on any ledger. The first wave that can consume the fallback Earlier automation moved the process and left the practitioners Earlier waves removed people too. Mainframes removed clerks, ERP removed accounting staff, self-service removed transaction processors. The distinction is not headcount. What those waves left behind was enough capability to reconstruct the process. The people who understood the work were still in the building, and parts of the job were still done by hand, so the understanding stayed current without anybody maintaining it. An AI deployment is funded on the basis that it absorbs volume. That is the case, it is usually a good case, and the savings land. What lands with them is the headcount change that made the numbers work. Some of those people leave. Others are redeployed, which sounds gentler and often is, though it depends on how completely and for how long they leave the work. The relevant question is not whether they are still on the payroll. It is whether anyone is still doing the thing by hand, and how recently. Keeping a human in the loop does not solve it What thirty years of research says about reliable systems The obvious answer is that the people are still there, supervising rather than doing. A human stays in the loop, the process gets reviewed, the capability is preserved. Two things are wrong with that, and both are established rather than argued. Reliability degrades the reviewer. Parasuraman, Molloy and Singh ran a multitask flight simulation in 1993. Participants monitored an automated system while handling other jobs. For one group the automation was consistently reliable. For the other it varied. After about twenty minutes, the group watching the reliable automation was substantially worse at catching its failures. The reliability itself degraded the monitor, and the effect disappears when monitoring is the only job, which matters because enterprise reviewers rarely have one job. Rarity degrades detection. Wolfe, Horowitz and Kenner published in Nature in 2005. Observers searched simulated baggage x-rays, and the only thing that changed between conditions was how often a target was actually there. At fifty percent prevalence they missed seven percent. At one percent they missed thirty. Think of it as Where’s Waldo, except Waldo is only on one page in a hundred. You would find him on page one. By page forty you would be flipping. By page ninety-nine you have stopped looking in any way that counts, and when he finally shows up you turn straight past him. Nobody decides to stop looking. The absence of a decision is the whole problem. The two research traditions rarely cite each other, and they describe the same condition from opposite ends. Ninety-nine percent accuracy, from the reviewer’s side of the desk, is an environment where an error shows up once in a hundred. So two things govern whether a reviewer is a control or a formality: how often errors appear, and whether the reviewer is doing anything else. Errors frequent Errors rare Reviewer does nothing else Works. Frequency keeps them calibrated, attention is undivided Prevalence pulls detection down, though single-task monitors held up regardless of reliability Reviewer has a queue and a day job Degraded and still functioning. Finding things regularly keeps them in the task Both effects compound. The quitting threshold falls and attention goes to the work that visibly needs it This is a way of locating the problem rather than a model of it. Bottom right is where many enterprise review roles end up, and it is where a successful deployment can put you. The deployment moves you right, because the errors the reviewer sees become rarer as the model improves. It moves you down, because the headcount that justified the project is