Authorities blamed shoddy construction as the probable cause for the collapse.
Rescuers searched for scores of people believed trapped in the building, located in a poor Lagos district, but it was not known how many were there when it collapsed Tuesday night. The building contained 36 apartments and up to 180 people, and there were several businesses on the ground floor.
Red Cross official Timothy Oladene said 36 people, many of them injured and caked with dust and debris, had been rescued so far and sent by ambulance to hospitals across the city. The bodies of two children, their faces covered in dirt and blood, were among those hauled out by rescue workers.
Lagos state Gov. Bola Tinubu visited the site and vowed to find and prosecute those responsible.
"The developer cannot run away. He has property in Lagos. The government will seize it and prosecute him," Tinubu told a gathering crowd.
Prince Oniru, an infrastructure adviser to the Lagos state government, said the building was weak and blamed those who built it. "The material used in the construction was very poor," he said.
The four-story structure was reduced to a two-story pile of rubble: a twisted mess from which wires and concrete beams protruded. Ripped mattresses, smashed sofas, children's books and broken plates littered a dusty, chaotic scene.
Several trapped people could be heard shouting from inside what was left of the building, witnesses said.
Oladene said power went out in the Ebute-Meta district Tuesday evening and volunteers worked though the night with candles and flashlights.
By Wednesday morning, police kept back thousands of people who crowded around the building looking for loved ones and picking through the rubble. The crowds cheered as a woman was brought out alive.
Oladene said a Red Cross employee was among those trapped inside.
"We have rescued his wife and two children but the man is still inside," Oladene said. "We will be here until we find the last person."
Three cranes worked to lift larger pieces of rubble.
Oladene expressed concern about security at the site and said his cell phone had already been stolen. Because crime in the city is so bad, most residents stay home at night.
"It's essential to rescue as many people as possible before dark," Oladene said, as police beat back people pressing forward to identify a body being hauled out.
Tears streamed down the faces of many in the crowd. One woman, 36-year-old Fatima Abdullah, said she was looking for her mother.
"I've been waiting all night," she said. "I can't find my mother."
One man, 37-year-old Olukayode Toyoba, said a hospital had denied his three injured children treatment until a neighbor offered to pay for their medical expenses. He owned a hair salon that was crushed, and was looking for his missing wife.
Another man, 34-year-old Tiawa Azeez, picked through the debris with a knife and a pair of pliers, trying to get rescue workers to dig up a section where he thought his brother had been buried.
"It's sinking and they are inside," he said, referring to the weight of some 100 people standing on top of the rubble.
In late March, the top of one of Lagos' tallest buildings slid off in a storm, and falling debris killed at least two people. About nine stories of the 21-storey Nigerian Industrial Development Bank crumbled and fell during a heavy downpour. There was a fire in the building two days before.