CBS New York reports that at least 62 others were also injured in the blast, which happened around 9:15 a.m. Wednesday on Park Avenue at 116th Street, sources said.
Speaking from the scene, Mayor Bill de Blasio said the explosion was caused by a gas leak. He said some of those unaccounted for might have fled to safety.
One of the buildings that collapsed had a piano repair shop with apartments above. The second building housed a church.
Officials believe both buildings were five stories.
WCBS reported at least three people, two of them women, were killed in the explosion. The fire department said more than 20 people were injured, including two with life-threatening injuries. The FBI said two agents were injured in the explosion but were expected to make a full recovery.
One of the women who was killed was identified as Sgt. Griselde Camacho, a public safety officer at New York City's Hunter College since 2008.
"Our hearts go out to Griselde's family at this terrible time," the school's president, Jennifer J. Raab, said in a statement.
The second was identified as Carmen Tanco, 67, who worked as a dental assistant.
The White House issued a statement offering "thoughts and prayers" and commending first responders. The National Transportation Safety Board said it was sending a team of experts to New York City to investigate.
Video from the scene showed huge flames and billowing smoke as hundreds of firefighters battled the blaze, which was upgraded to five-alarms. Debris and rubble could be seen scattered all over the area.
Witnesses reported hearing a loud explosion.
"I was home and then next thing you know I'm laying down and I hear this big explosion. I came out, looked up and a building was just gone," eyewitness Mack Mayor told CBS 2. "It's just like a war zone out here."
"The smoke started to rise. It looked like something fell because it wasn't like a fire. It just looked like debris smoke, similar to 9/11," said witness Samuel Paul, who was on the 15th floor of a building on 125th Street.
Con Edison said crews were responding to the area on a report of a gas odor just prior to the explosion.
"The call came in at 9:13 a.m. from a resident who reported smelling gas inside the apartment, but the resident indicated the odor may have been coming from outside the building," Con Ed spokesman Sydney Alvarez said. "Two Con Edison crews were dispatched at about 9:15 a.m. and arrived just after the explosion occurred."
Alvarez said the street was served by an eight-inch low-pressure gas main and that crews were checking gas lines and other structures for any possible leaks.
De Blasio said the "only indication of danger" came about 15 minutes before the blast.
Carmen Vargas-Rosa, who owns the church, said there was a gas odor nearby Tuesday night.
"Last night when we came out of church, at the corner outside in the street there was a smell of gas," she said. "It would come and go, so I thought it was from the corner store there so I went to the owners and told them and they said they would look into it and then we went home."
Vargas-Rosa said four of her tenants are unaccounted for.
"One of them was off from work so most likely she was in the building," Vargas-Rosa said. "We have a mother who has a sick son, so unless they were at an appointment in the hospital, they may have been in the explosion and then we have a young man and his wife who go to school late so they may have been sleeping and we can't reach them."
The New York Police Department ruled out terrorism as a cause of the explosion, but the bomb squad and the FBI were both on the scene.
Metro-North said the explosion happened adjacent to their tracks. Service in and out of Grand Central Terminal was suspended until further notice.
Many took to Twitter and began posting pictures of smoke rising from the area.