Bad Data: How Quants Convinced Roche to Compromise Patient Safety

Roche (ROG.VX) has been sued by so many patients who took its acne drug Accutane -- as many as 5,000 by some estimates -- that it stopped marketing the drug in 2009. Hundreds of those cases allege that the drug causes irritable bowel disease, an irreversible condition that can lead to Crohn's disease and, eventually, having parts of your colon removed. Unlucky patients can end up dependent on colostomy bags.

Yet according to a New Jersey state appeals court ruling, a Roche executive discussed Accutane's bowel problems with the FDA as early as 1978. Before the drug was approved by the FDA in 1982, a Roche study found 21.6 percent of patients on the drug experienced gastrointestinal side effects. Still, the drug was approved without a warning for bowel problems. Roche did not admit in its warnings for the drug that there was a link between Accutane and bowel disorders until 2000, the ruling states. (The ruling remands the case for a new trial.)

Why did managers at Roche stay silent on Accutane's risks for so long? One reason, the ruling hints, is that the company became too dependent on its own quantitative statistical analyses of the drug's adverse events. Those analyses told Roche that even though it received hundreds of reports of serious bowel problems in the 1980s and 1990s they were not statistically significant compared to the rates of reports in the general population. While the databases may indeed have shown that to be true, there's a difference between collecting good data (what quants do) and applying good judgment to data (what good managers do).

Most importantly, managers need to realize that their decisions usually occur with incomplete information, which is what Roche had. The ruling quotes a memo from one of the company's own doctors noting that IBD cases reported to Roche are probably underreported, and it's known that cases of IBD generally are underreported. The ruling also quotes a 1983 letter from Public Citizen to the FDA seeking enhanced warnings for Accutane, including for bowel problems, which mentions that only about one in 10 adverse events are ever reported to the companies that make the drugs.

Despite this underreporting, Roche continued to compare the cases it had on hand with the total number of prescriptions it had sold -- vastly underrepresenting the probable level of bowel problems among all patients. In addition, Roche used an algorithm to classify adverse event reports. The algorithm assigned reports to one of four categories, from "highly probable" to "doubtful." While it's true that not all reported adverse events are caused by the drug in question, you can see that Roche's system automatically discounts a certain portion of the already underreported level of adverse events.

Roche consulted its database repeatedly from 1982 through 2000, and concluded that bowel problems were not statistically significant among Accutane users. In 1984 the company sent a "dear doctor" letter warning physicians to be watchful with Accutane patients, but that letter said "a precise cause and effect relationship has not been shown."

Another "Dear Doctor" letter in 1998 maintained that, based on the available data, Accutane "does not cause" IBD.

Eventually, in 2000, Roche admitted that Accutane may cause IBD.

Roche's managers made a key error often committed by quants: They let the data make the decision for them instead of using the data to make their own decision. While the data may not have indicated a high rate of bowel problems, Roche should have also taken into account several non-quantitative factors: such problems are greatly underreported; that the dermatology doctors prescribing the drug were not experts in bowel disorders; and that Accutane is incredibly poisonous (you can't take it while pregnant). In addition, some good old-fashioned non-quantitative common sense might have helped: If IBD was not significant, why were the FDA, Public Citizen and thousands of Roche's own patients bothering to complain about it?

Related:

Image by Flickr user bionic teaching, CC.

More from CBS News