A study of thirty years of information disasters, from Three Mile Island to the Invasion of Iraq shows a common characteristic: alarms were silenced in all cases. And it happened again on Wall Street as the value of risky investments began to collapse. Scott Hansel, who has covered Wall Street for 13 years, recently reported in the New York Times: “The people who ran the financial firms chose to program their risk-management systems with overly optimistic assumptions and to feed them oversimplified data. This kept them from sounding the alarm early enough.”

Financial firms are required by federal regulations to monitor their positions, and if risk rises above a specified level they are required to reduce their bets or set aside more capital. That is the law. But the risk models were fed false data, and the sample of risks monitored was intentionally skewed. “There was a willful designing of the systems to measure the risks in a certain way that would not necessarily pick up all the right risks,” said Gregg Berman, the co-head of the risk-management group at RiskMetrics, a software company spun out of JPMorgan. The alarms never went off. In a period of six months, Lehman Brothers, Bear Stearns, Merrill Lynch, and AIG together lost about $300 billion in market value. By late September, 2008, the federal government was forced to wade in with $700 billion in order to keep the credit market working.

Silencing the alarms has a long and lugubrious history. In 1979, at the Three Mile Island nuclear power plant, the red light designed to tell whether a critical valve was stuck open had been wired to the switch, not to the valve, and while the water was rushing out of the reactor, the million dollar dashboard showed that everything was fine. In 1983, when the pilot of Korean Air Lines Flight 007 keyed the wrong coordinates into the onboard computer, the alarm started to sound. He turned off the alarm system. The plane strayed into Russian airspace and was shot down. In 1983, when two NASA engineers warned that the O-ring on Shuttle Challenger would burn through at low temperatures, they were told by management to keep their mouths shut.

In August 2001, when a CIA briefer told President Bush that bin Laden was “determined to strike in the US,” Bush told him, “well you've covered your ass now” and spent the rest of the day fishing. In 2002, when the White House showed Senator Daschle the satellite photos of Iraqi “weapons of mass destruction” Daschle, who had been trained as a satellite photo analyst in the Air Force, said the pictures were too blurry to be indicative. But VP Cheney asked him to keep his concerns to himself, and he did. In 2007, when a kidney transplant patient's prescription was wrong transcribed, the pharmacy computer's high dose alert went off, but the clerk ignored the alarm and the patient nearly died.

It is always difficult to heed warnings. They confront us with uncertain information about a possibility we don't want to think about. They are often presented by experts who talk funny and cite complicated data. Automatic alarms are particularly dangerous because there is always the possibility of a technical malfunction. Home burglar alarms ringing at the police station are false 97 percent of the time, according to William Bratton, Los Angeles chief of police, and from now on, he said, they will be ignored unless “someone can prove that there is a genuine emergency.”

As a species we prefer to hope and, in the case of the financial services sector over the last few years, we go out of the way to fool ourselves.

(Originally published at GoArticles and reprinted with permission from the author, Christopher Burns).