At least 13 people were injured. The cause of the fire was not immediately known.
Gov. Arnold Schwarzenegger has proclaimed a state of emergency in Santa Barbara County.
The governor's declaration on Friday dedicates state personnel and equipment to the firefighting effort. It also says the Federal Emergency Management Agency has agreed to give California a grant to help fight the fire.
The blaze broke out just before 6 p.m. Thursday and spread to about 2,500 acres - nearly 4 square miles - by early Friday, destroying dozens of luxury homes and parts of a college campus in the foothills of Montecito, just southeast of Santa Barbara.
CBS News correspondent Hattie Kauffman reports that many mansions that dot the landscape were quickly engulfed in flames as the wind-whipped fire swirled uncontrollably.
"I saw $15 million in houses burn, without a doubt," Santa Barbara evacuee Tom Bain said. "They were just blowing up. It was really, intensely hot."
Within less than six hours, it destroyed dozens of luxury homes and several buildings on the campus of Westmont College, a Christian liberal arts college nestled amid wooded rolling hills, where some 1,000 students were caught off-guard by the rapidly moving flames.
"It came pretty fast," said Tyler Rollema, a 19-year-old sophomore who was eating dinner in the cafeteria when students were told to head to the gym. "We came out and it was just blazing."
Fire officials began an aggressive attack from the air at daybreak with the help of nine water-dropping helicopters and 10 air tankers, said Terri Nisich, a spokeswoman with the Santa Barbara County Executive Office. A high wind warning was in effect in Los Angeles and Ventura counties through Saturday, with possible gusts of up to 70 mph in some passes and canyons.
Earlier, Nicole Koon, another county spokeswoman, said about 5,400 homes were evacuated in tony Montecito, which has 14,000 residents and counts celebrities like Oprah Winfrey and Michael Douglas as homeowners.
She had no immediate detail on the damage and evacuations in neighboring Santa Barbara.
By Friday morning, the city's downtown was filled with dense, acrid smoke and people walked on the streets with towels and masks over their faces.
Michele Mickiewicz, a spokeswoman with the county emergency operations center, said Friday that 10 people were treated for smoke inhalation and three had burn injuries. Earlier, Santa Barbara Cottage Hospital reported receiving two patients with substantial burns.
Thousands of feet above the flames, footage shot from television helicopters showed what initially looked like a massive campfire with dozens of glowing embers. When cameras zoomed in, however, what appeared to be flaring coals turned out to be houses - many of them sprawling estates - gutted by flame. Palm trees were lit like burning matches.
With Santa Ana winds gusting over 50 miles an hour, keeping the water-dropping helicopters grounded for hours, it was nearly impossible for firefighters on the ground to attack the blaze.
About 500 firefighters are racing to stop the flames from spreading west to Santa Barbara and downhill toward more homes.
KCAL's helicopter crew, reporting from over the blaze, said the fire was jumping from house to house, in some cases right over the heads of firefighting crews trying desperately to contain it.
From the air, you could see home after home after home engulfed in flames.
"Absolute devastation," said one pilot.
"It looked like lava coming down a volcano," Leslie Hollis Lopez said as she gathered belongings from her house.
Tom Bain, a 54-year-old electrician, said authorities ordered him to leave his home in Santa Barbara around midnight.
Bain quickly collected his three cats, his work files and his computer and was out his house within five minutes. On his way out, he saw at least six mansions on a ridge above his home explode into flames and the cool night air was warmed to about 80 degrees by the fire.
"You can just hear the explosions ... of vehicles, homes," Michaelo Rosso told CBS affiliate KCAL-TV as he prepared to leave his home. "It sounds like the Fourth of July out here."
About 200 people spent the night at an evacuation center at a high school in nearby Goleta, but rest was out of the question for Ed Naha. He was worried about his home in the hills above Santa Barbara.
"I don't think we are going to have the house when we go back," Naha said.
The 58-year-old writer had been home working on his computer when smoke blanketed his house. He gathered his insurance documents, his wife and two dogs and left as flames approached his neighborhood.
"We are used to seeing smoke because we do have fires up here, but I've never seen that reddish, hellish glow that close," he said. "I was waiting for Dante and Virgil to show up."
At Westmont College, the air was dense with smoke and the scent of burning pine. Students holed up in the school's gym as flames chewed through a eucalyptus grove on the 135-acre campus and destroyed several buildings housing the physics and psychology departments, a dormitory and at least one faculty home, college spokesman Scott Craig said.
"I saw flames about 100 feet high in the air shooting up with the wind just howling," he told AP Radio. "Now when the wind howls and you've got palm trees and eucalyptus trees that are literally exploding with their hot oil, you've got these big, red hot embers that are flying through the sky and are catching anything on fire."
"I was terrified," one woman told CBS News. "The gym was filled with smoke. The smoke was coming through the vents. And we could see flames right outside."
Beth Lazor, 18, said she was in her dorm when the alarm went off. She said she only had time to grab her laptop, phone, a teddy bear and a debit card before fleeing the burning building.
Her roommate, Catherine Wilson, said she didn't have time to get anything.
"I came out and the whole hill was glowing," Wilson said. "There were embers falling down."
The college said on its Web site it was considering moving the students to a Red Cross shelter if they can get safely through the fire.
The winds weakened overnight, with gusts reaching from 17 to 25 mph, said Jamie Meier, a meteorologist with the National Weather Service in Oxnard. "We're expecting conditions to improve for firefighters on the lines, but it will still be warm and dry through tomorrow," she said.
The fire temporarily knocked out power to more than 20,000 homes in Santa Barbara, Southern California Edison spokesman Paul Klein said
A red flag warning for Los Angeles and Ventura counties is in effect from 8 a.m. today to 4 p.m. tomorrow.
Firefighters faced wind gusts as high as 70 mph Thursday night. Gusts were expected to remain strong through early Friday, according to the National Weather Service in Oxnard. The fierce winds (known locally as "sundowners") blow from land to sea in the evening, reversing the normal onshore flow of cool, moist sea breezes. They are caused by the area's unique topography.
Montecito and its multimillion-dollar homes with ocean views have long attracted celebrities such as Michael Douglas, Robe Lowe and Oprah Winfrey, who owns a 42-acre estate there. The landmark Montecito Inn was built in the 1920s by Charlie Chaplin and Fatty Arbuckle, and the nearby San Ysidro Ranch was the honeymoon site of John F. Kennedy in 1953.
Publicists for Lowe and Winfrey told the AP the celebrities' homes had not been destroyed and that neither was staying in the area Thursday night.
Montecito suffered a major fire in 1977, when more than 200 homes burned.