The force of the collision knocked the van's engine into the passenger compartment and sent flames as high as the top of a nearby telephone pole into the air, authorities said. Those killed in the crash were all men, ages 55 to 79.
"They were being burned alive," said Michael Hickey, who was driving by and stopped to help rescue those inside. "I could tell they were conscious because they were moving around."
Five other people in the van, including the driver, Guy Thelemaque, were injured. All were in serious condition.
Officials said the van was speeding, that the driver had a suspended license and that no one in the van was wearing seat belts.
The New York Times reported that the adult home, Brooklyn Manor, was the scene of a long string of abuses, and that the state had repeatedly attempted to shut down or take over the facility.
The state has said the violations included unsanitary conditions, shoddy record-keeping and poor supervision of residents.
Passersby helped to rescue several men trapped in the van, but the flames were too intense to rescue all of the occupants.
The crash happened at about 3 p.m. on Wednesday in New York City's borough of Queens.
"We are still investigating whether this is just a tragic accident or something that involves some criminality," said Queens District Attorney Richard Brown. He said the van was traveling at "a high rate of speed."