Super Bowl blackout caused by device installed to prevent power outage
Corey Graham #24, Ray Lewis #52 and Paul Kruger #99 of the Baltimore Ravens walk towards the sideline as play was suspended in the third quarter for 34 minutes due to a power outage against the San Francisco 49ers during Super Bowl XLVII at the Mercedes-Benz Superdome on February 3, 2013 in New Orleans, Louisiana. / Photo by Christian Petersen/Getty Images
NEW ORLEANS An electrical device that had been installed expressly to prevent a power outage caused the Super Bowl blackout, the stadium's power company said Friday as it took the blame for the outage that brought the game to a halt for more than a half-hour.
Super Bowl blackout: Memo reveals officials worried about failure
Following the announcement, the manufacturer of the device, known as a relay, released a statement implying that the problem was not with the part but with how it was used.
Officials of Entergy New Orleans, a subsidiary of New Orleans-based Entergy Corp., said the device, called a relay, had been installed in switching gear to protect the Superdome from a cable failure between the company's incoming power line and lines that run into the stadium.
The switching gear is housed in a building known as "the vault" near the stadium that receives a line directly from a nearby Entergy power substation. Once the line reaches the vault, it splits into two cables that go into the Superdome.
- Documents: Worries about outage before Super Bowl
- Super Bowl power outage: Outside experts to investigate cause
- Behind the scenes in the NFL control room during Super Bowl blackout
Company officials said the device performed with no problems during January's Sugar Bowl and other earlier events, but has been removed and will be replaced. All systems at the Superdome are now working and the dome will host a major Mardi Gras event Saturday night, said Doug Thornton, an executive with SMG, the company that manages the stadium for the state.
The power failure at Sunday's big game cut lights to about half of the stadium for 34 minutes, halting play between the Baltimore Ravens and San Francisco 49ers.
The FBI had ruled out cyberterrorism as a cause.
Entergy's announcement came shortly before officials appeared before a committee of the City Council, which is the regulatory body for the company, to answer questions about the outage.
Not long after the power company's announcement, the manufacturer, Chicago-based S&C Electric Company, released a statement saying that the power outage occurred because the electric load was too much for the equipment, "exceeding the trip setting for the switchgear relay as set by the system operators."
"Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power," said Michael J.S. Edmonds, vice president of strategic solutions for the company.
- Complete coverage of Super Bowl XLVII on CBSSports.com
- CBSNews.com: Complete coverage of Super Bowl XLVII
The statement did not name the "system operators," but the equipment was owned and installed by Entergy New Orleans.
Entergy officials could not immediately be reached for comment after the statement was released Friday.
Entergy New Orleans CEO Charles Rice and Dennis Dawsey, an Entergy vice president for distribution, told the City Council that SMG agrees the cause of the outage was a relay failure. Asked if the two corporations still plan to hire a third-party investigator, Rice said that possibility remains open.
Committee member Jackie Clarkson pressed for such an independent probe. "We've told the public we're going to have an outside investigation," she said.
"We'll work closely with SMG and if there is a need for a third-party investigation, we will do that," Rice said, adding that Entergy was also working with the manufacturer.
Shabab Mehraeen, an assistant professor of electrical engineering at Louisiana State University, said the relay device is a common electrical fixture in businesses and massive facilities such as the Superdome.
"They are designed to keep a problem they sense from becoming something bigger, like a fire or catastrophic event," said Mehraeen, who holds a doctorate from the Missouri University of Science and Technology in Rolla, Mo.
The devices vary in size, and while Mehraeen noted he was not familiar with the specifics of the relay at the Superdome, he added, "I wouldn't be surprised if it was bigger than a truck."
Mehraeen said the reasons the devices fail are the subject of much academic research into the interaction of relays with the complex electrical systems they regulate.
"It's not unusual for them to have problems," he said. "They can be unpredictable despite national testing standards recommended by manufacturers."
Entergy and SMG had both upgraded lines and equipment in the months leading up to the Super Bowl. Rice said the new switching gear, with the faulty relay, was installed as part of a $4.2 million upgrade by Entergy, including the installation of a new power line dedicated solely to the stadium.
Earlier Friday, Entergy said the device was the cause of the power outage.
In a separate project, SMG replaced lines coming into the stadium after managers expressed concerns the Superdome might be vulnerable to a power failure like the one that struck Candlestick Park during a 49ers Monday Night Football game in 2011. That outage was blamed at least partly on a transformer explosion.
Thornton stressed Friday that the dome was drawing only about two-thirds of its power capacity Super Bowl night, and said typical NFL games in late August or September can draw a little more.
City officials had worried that the Super Bowl outage might harm New Orleans' chances of getting another NFL championship game.
But NFL Commissioner Roger Goodell downplayed that possibility after the outage, saying the NFL planned to keep New Orleans in its Super Bowl plans. Mayor Mitch Landrieu said the city intends to bid for the Super Bowl in 2018.
Fans watching from home weren't deterred, either. An estimated 108.4 million television viewers saw the Baltimore Ravens beat the San Francisco 49ers 34-31, making it the third-most-viewed program in television history. Both the 2010 and 2011 games hit the 111 million mark.
Popular in Sports
- Sergio Garcia apologizes to Tiger Woods for "fried chicken" remark
- Paralyzed Tulane player to spend holidays at home
- Home of Nuggets' Andersen searched, items seized
- Study: College athletes denied $6.2B over four years
- NFL's Top 100 Players of All-Time: Debate
- Watch: 7-foot-5 teen basketball player dominates
- "Egyptian Popeye" says 31-inch arms all-natural
- MLB Settles On Steroid Standards
I don't know this for a fact, but I would bet that the switchgear that feeds the stadium is old. I would bet that it's very old, in fact. And...we all know that it was exposed to water in the not-too-distant past! It's not uncommon for any power company in the U.S. (or the world!) to have many components of their power infrastructure be at least 50 years old or more. Utilities have regular discussions about their "aging infrastructure" and what to do about it. I won't go into this subject at great length, but utilities can't upgrade everything on their system as they get old because the cost to upgrade, revamp or replace this type of equipment is very costly. Problematic equipment, of course, gets priority when it comes to upgrading or replacing equipment.
Some old equipment has proven to be more reliable and requires less maintenance than some of the newer replacement equipment. This is not always the case, fortunately, because depending on the age of this equipment--including relays--utilities may not be able to get parts or equivalent replacements anymore. The relay manufacturers may have stopped making that particular model and may have depleted the stock from their shelves years ago.
So what could have happened with this relay issue at the stadium? Let me first say that I know for a fact that utilities go to extra-great lengths to do everything in their abilities to keep the power flowing at events like the Super Bowl. They go over everything associated with the power equipment that feeds the stadium and discuss contingency power plans ahead of time. They will also make sure they have extra staff onboard during these events to cover any problems that may arise. The fact that the utility replaced a relay before the game with the intention of increasing reliability proves that what I'm saying is true. Utilities don't go around replacing relays at random...they will have a reason.
My guess is that they either noticed something visually wrong with the original relay or perhaps it wasn't responding as it should when they tested the relay during normal periodic maintenance. Or they may have decided to test this relay prior to the game just for that event. Regardless, they didn't feel comfortable with that relay for some reason and replaced it. The utility should not be criticized for doing something like this--it's the responsible thing to do and any rational customer would expect it.
The relay in question is part of a "protective relay scheme" and its purpose is to ultimately isolate faults on the power system so that damage isn't done to more parts of the system than is necessary and to also prevent customers from unnecessarily losing their power. This protective relay scheme is highly critical and is of major importance in any utility's power system. Without it, we would all experience more frequent and more sustained power outages for sure.
Assuming the original relay was very old (and had previous water exposure), if the utility replaced it with an exact replacement from their personal stock or was lucky enough to obtain one from the relay manufacturer, that replacement relay is most likely just as old as the original.
If an identical replacement relay could not be obtained, you can sometimes use another model relay and modify it in some way so that it has the same (or similar) characteristics. This is not the preferred choice and it could be risky, but in a pinch, it may be the best temporary option.
The relay technician who installed the relay could have inadvertently applied the wrong settings to the relay, which could have made the relay more "sensitive" and caused a power outage prematurely. This may sound pathetic, but these are human technicians. How many times have we done something foolish because we had other things on our minds at the time and wasn't paying enough attention to what we were doing?
There are probably at least a dozen other possibilities that could have caused the Super Bowl power outage, but it's clear to me that the utility was thinking about ways to make the power to the stadium more reliable. I know first hand what goes into the electric reliability preparation for big events. No one in that business wants egg on their face and they absolutely go to extreme lengths to cover all the bases.
Sorry, I was channelling Glenn Beck for a moment there.