These steel monsters known as blowout preventers or BOPs - sometimes as big as a double-decker bus and weighing up to 640,000 pounds - guard the mouth of wells. They act as the last defense to choke off unintended releases, slamming a gushing pipe with up to 1 million pounds of force.
While the precise causes of the April 20 explosion and spill remain unknown, investigators are focusing on the blowout preventer on the Deepwater Horizon rig operated by BP PLC as one likely contributor.
To hear some industry officials talk, these devices are virtually foolproof.
But a detailed AP review shows that reliability questions have long shadowed blowout preventers:
• Accident reports from the U.S. Minerals Management Service, a branch of the Interior Department, show that the devices have failed or otherwise played a role in at least 14 accidents, mostly since 2005.
• Government and industry reports have raised questions about the reliability of blowout preventers for more than a decade. A 2003 report by Transocean, the owner of the destroyed rig, said: "Floating drilling rig downtime due to poor BOP reliability is a common and very costly issue confronting all offshore drilling contractors."
• Lawsuits have fingered these valves as a factor in previous blowouts.
It is unclear why the blowout valves on the Deepwater Horizon didn't stop the April 20 blast that killed 11 workers and has sent millions of gallons of oil spewing into Gulf. Interviews with rig workers conducted as part of BP's internal investigation into the explosion indicate that a methane gas bubble escaped from the well and expanded quickly as it shot up the drill column
Since then, the minerals agency has been inspecting offshore rigs and platforms to verify testing of these valves and check emergency exercises. On Friday, a senior agency official told the AP that regulators had been comfortable that the valves were reliable - until the blowout.
"Based on the record, we have felt that these were performing the job they were supposed to perform," Deputy Director Walter Cruickshank said. "This incident is going to make us re-examine that assumption."
He said new procedures and rules may be needed, including certifying blowout preventers by an independent group of experts. He also said the agency may revise its peeled-back testing requirement of 1998, when it replaced a weekly regimen with biweekly pressure tests.
Congress plans hearings that will consider BOP reliability. "The safety valve is not so safe," said U.S. Sen. Maria Cantwell, D-Wash. She said the industry knew this kind of part sometimes fails, but it acted as if it couldn't.
After the accident, BP CEO Tony Hayward said of blowout preventers in general: "It's unprecedented for it to fail."
Yet the AP review turned up instances where preventer seals have failed outright, obstructions have blocked them, or valves simply weren't designed for the task. Sometimes there were blowouts.
The control systems also have proved goof-prone. When a worker accidentally disconnected a blowout preventer at one rig in 2000, federal regulators recommended changes in the control panels. Later that year, a worker at a rig off the Louisiana coast was making those very changes when he accidentally pushed the wrong button - and unlatched the valves; the ensuing blowout released 8,400 gallons of crude.
The government has long known of such problems, according to a historical review conducted by the AP. In the late 1990s, the industry appealed for fewer required pressure tests on these valves. The federal minerals service did two studies, each finding that failures were more common than the industry said.
But the agency, known as MMS, then did its turnaround and required tests half as often. It estimated that the rule would yield an annual savings of up to $340,000 per rig. An industry executive praised the "flexibility" of regulators, long plagued with accusations that it has been too cozy with the industry it supervises.
Laurence Power, of Robert Gordon University in Aberdeen, Scotland, an engineering teacher who has studied these valves in offshore oil wells, said he has "not been able to see their logic" for reducing the frequency of testing.
In 1999, right after that rule change, an MMS-commissioned report by a research group identified 117 blowout preventer failures at deepwater rigs within the previous year. These breakdowns created 3,638 hours of lost time - a 4 percent chunk of drilling time.
In 2004, an engineering study for federal regulators said only 3 of 14 new devices could shear pipe, as sometimes required to check leaks, at maximum rated depths. Only half of operators accepting a newly built device tested this function during commissioning or acceptance, according to the report.
"This grim snapshot illustrates the lack of preparedness in the industry to shear and seal a well with the last line of defense against a blowout," the report warned.
Two years later, a trade journal's article still noted that shearing preventers "may also have difficulty cutting today's high-strength, high toughness drill pipe" at deep wells.
The special cutting preventers were blamed in 1979 for the biggest peacetime well spill in history, when about 140 million gallons of oil poured from a Mexican well in the Gulf.
Questions about reliability hung heavily but were mostly unspoken Thursday at a Houston conference on offshore oil rig technology. Shown a spreadsheet of problems with blowout preventers, Transocean technology manager John Kozicz said, "We know that - but they don't happen frequently."
Even Transocean's Earl Shanks, lead author of the 2003 study reporting "poor BOP reliability," now views blowout preventers as "very reliable." But he did acknowledge problems in the complex electronic and hydraulic tangle that activates and controls the devices. At Deepwater Horizon, he said, "Something went wrong - and we don't know what."
Cameron International, which made the Deepwater Horizon preventers, has acknowledged that these lumbering emergency stoppers need lots of upkeep. "You have to maintain it," CEO Jack Moore told investors last year. "You have to replace the mechanical and rubber elements."
Cameron International did not respond to AP questions about reliability. But it has had to face such questions in court.
A 2008 federal lawsuit claims its faulty blowout preventers contributed to a well blowout. The suit makes the same claim about other valves installed at the rig but made by Hydril.
A Hydril Pressure Control representative said he couldn't be quoted by name under company policy, but he defended the safety of his company's preventers. Asked about the lawsuit, he said, "It is a matter of litigation, and we have denied the allegation and strongly believe in the merits of our case."
More Coverage of Gulf Oil Spill:
Oil Leak Container Touches Down on Seafloor
Pelicans' Brief Success Threatened by Oil Spill
Oil Washing Ashore at Island Off Louisiana Coast
Marine Food Chain Seen at Risk After Oil Spill