A sixth person, a firefighter, was injured during the rescue efforts after the 9 a.m. collapse.
"I stepped out of my building and I heard a loud boom," said Racquel Gayle, who said she was about half a block from the site. "Everything rocked. I thought it was a terrorist bomb; people were running everywhere."
She said she saw people pull a baby out of the debris, "but I stood frozen in fear. I've never seen anything like this."
The former Gristedes supermarket was being demolished to make room for a high-rise building when part of the remaining structure collapsed, Fire Department spokesman Frank Gribbon said. The 25 construction workers at the site were accounted for, he said.
It was not immediately clear how the child was injured, officials said, but some debris spilled onto the sidewalk and street. The victims were taken to a hospital. Gribbon said most suffered broken bones.
Gristedes spokesman Gerald McKelvey said the building was closed in April and sold to the Carlyle Group. Some neighborhood residents have protested plans to build a skyscraper there.
At least 30 fire department units responded to the collapse, said Fire Department spokesman Charlie Markey. Dozens of firefighters sifted through the rubble and collapsed scaffolding around the building looking for victims.
The building is at 100th Street and Broadway on the Upper West Side of Manhattan.