In the study, Zetia failed to shrink buildups in artery walls while a rival drug, Niaspan, did so significantly. Zetia users also suffered more heart attacks and other problems although the numbers of these events were too small to draw firm conclusions.
Zetia "has been on the market for about seven years and we still haven't proven that it improves clinical outcomes," said Dr. Roger Blumenthal, preventive cardiology chief at Johns Hopkins University. The new results will be "very influential" in getting more doctors to turn to Niaspan, he said.
He wrote an editorial accompanying the results, which were presented Sunday at an American Heart Association conference and published on the Internet by the New England Journal of Medicine.
Despite the study's findings, CBS News medical correspondent Dr. Jennifer Ashton advises cholesterol drug users not to stop taking their medications before talking with their physician.
Statins such as Lipitor and Crestor have long been used to lower LDL, or bad cholesterol, and are known to cut the risk of heart problems. Nevertheless, many statin users still suffer heart attacks, so doctors have been testing adding a second medicine to further lower risk.
One they are trying is Niaspan, a slow-release version of niacin, a type of B vitamin that raises HDL, or good cholesterol. Another is Zetia, which lowers bad cholesterol in a different way than statins do, by blocking its absorption in the gut.
Vytorin is a pill that combines Zetia with a statin. Both are sold by Merck & Co. of Whitehouse Station, N.J. Niaspan is made by North Chicago, Ill.-based Abbott Laboratories. All three of these drugs cost between $3 and $4 a day, though niacin has been sold as a cheap generic for decades. The heart association advises consumers against using over-the-counter vitamin versions in place of prescriptions, though, because they may have purity and potency issues and other risks.
The new study was sponsored by Abbott, and several study leaders have been paid speakers or consultants to the company or to rival drugmakers.
Researchers enrolled 363 people with heart disease or a high risk for it who had been taking statins for six years on average. Many were from Walter Reed Army Medical Center, where study leader Dr. Allen Taylor formerly worked.
Half were given Niaspan and the rest, Zetia. Researchers stopped the study in June, after 208 participants had been on the medicines for 14 months, because one group was faring much better than the other.
Ultrasound images of neck arteries showed that Niaspan shrank buildups by about 2 percent, while Zetia had no effect on this even though it lowered bad cholesterol as expected.
There were two heart attacks, heart-related deaths or other heart-related problems in the 160 people given Niaspan, and nine among the 165 on Zetia.
"It should be better for the arteries and it wasn't," Taylor said of Zetia. "The drug wasn't operating as you otherwise would expect it to," raising concern that its effects are not fully understood, he said.
Merck's research chief, Peter Kim, said the study and number of heart attacks and other problems are too small to be conclusive. More than 25,000 people are in studies testing Zetia now, and independent monitors have found no problems that would lead them to stop the trials.
"We stand behind the safety of this drug," he said.
Kenneth Frazier, Merck's president of global human health, said: "The question is, how important is the study?" Shrinking plaque is no guarantee that heart attacks will be prevented, and not all studies have found a correlation, he said.
However, the difference in plaque that Niaspan made in this study "is precisely the same as the difference" that earlier studies found from statins, which are now known to save lives, said Dr. James Stein of the University of Wisconsin-Madison. He is a leading researcher on imaging artery buildups and in the past consulted for Schering-Plough Corp., which used to market Vytorin with Merck.
Dr. Sidney Smith, a former heart association president from the University of North Carolina at Chapel Hill, said that for people with newly diagnosed cholesterol problems, "the foundation is still statins." The new study "affirms the benefits of niacin" and makes it the next drug to add if people need more help, he said.
Dr. Anthony DeMaria, a past president of the American College of Cardiology from the University of California at San Diego, said the study "doesn't quite put the nail in the coffin" for Zetia, but pushes it way down on the list of things to try.
Last year, a large study found that the combo pill Vytorin was no more effective than Zocor alone, a statin now available as a cheap generic. In August, Merck and Schering-Plough agreed to pay $41.5 million to settle lawsuits claiming they delayed unfavorable study results on the drugs because they would hurt sales.
The drugs remain blockbusters: Vytorin had nearly $2 billion in sales in the United States in 2008; Zetia, more than $1.5 billion, according to IMS Health, a health care information and consulting company.
However, that's down substantially: Zetia prescriptions fell 22 percent, from nearly 16.5 million in 2007 to less than 13 million in 2008. Vytorin fell 24 percent in that time, from about 22 million in 2007 to 16.5 million in 2008. Vytorin sales were down another 36 percent in the first half of this year.
Niaspan has been gaining but lags far behind - 5.8 million prescriptions in 2008, up 11 percent from 2007.
Its main drawback is a prickly hot sensation called flushing that many people find intolerable. The extended-release version is supposed to minimize this, but a third of study participants still suffered it. The problem tends to go away with longer use and can be blunted by taking the medicine with aspirin, at bedtime, or with a low-fat snack, doctors say.