The crane split into pieces as it fell, pulverizing a four-story brownstone and demolishing parts of three other buildings.
Mayor Michael Bloomberg said at least four people, believed to be construction workers, have died and at least 10 people were injured in one of the city's worst construction accidents in recent memory.
"It is a tragic event," Bloomberg said.
The collapse created a virtual war zone on an affluent block on Manhattan's East Side: Cars were overturned and crushed. A huge dust cloud rose over the neighborhood. Rubble was piled several stories high.
"It's a horrible situation, very gory. There's blood in the street," said Lt. Gov. David Paterson, who takes over as governor for disgraced Eliot Spitzer on Monday.
An intensive rescue operation was under way to find anyone possibly trapped. One man was pulled from a collapsed townhouse 3 1/2 hours after the building was crushed.
Fire Commissioner Nicholas Scoppetta said the rescue was "a painstaking hand operation, as we try to remove the rubble so we don't cause further collapse or injure anyone who may still be in that building." He said the operation would continue all night if necessary, including the use of search dogs and thermal-imaging and listening devices.
The accident happened on 51st Street near 2nd Avenue at about 2:20 p.m. at the construction site of a high-rise condominium. It involved a crane that had been the subject of recent safety complaints.
About 19 of the building's 44 planned stories had been erected, and the crane was scheduled to be moved Saturday so workers could start work on a fresh story when a piece of steel fell and sheared off one of the ties holding it to the building, according to Stephen Kaplan, an owner of the Reliance Construction Group, which manages construction at the site.
"It was an absolute freak accident," Kaplan said. "All the piece of steel had to do was fall slightly left or right, and nothing would have happened."
Ben Galati, a 54-year-old doorman a high-rise apartment tower across the street, said he was in the basement when it happened, and ran for his life when he heard the structure smash into his building.
"I heard a rumble outside. I said, 'Let's get out of here! And then the crane came down. A split second later, I heard an explosion," he said.
John LaGreco, who owns Fubar located in the building that was crushed, said he feared one of his employees was dead in the rubble.
"Our bar is done," he said. "The crane crashed the whole building. If I wasn't watching a Yankees game, I would've come to work early and gotten killed."
The crane split into pieces as it fell. Part of it came to rest against an apartment tower, buckling its facade and smashing it upper floors. That building and others in the area were evacuated. Another piece of the crane hit traveled another half a block, ripping away walls and ceilings and crushing a small building.
Maureen Shey, who lives diagonal from the building where the crane collapsed, said she was on the phone laying on her bed when she saw the giant white crane heading straight for her windows.
"I heard a big crash, and I saw dust immediately. Bricks were flying through the air. I saw the whole thing. I thought the crane was coming in my window," she said.
The crane wound up missing her building, but others weren't as lucky.
Video broadcast by television news helicopters showed firefighters clambering through piles of rubble, several stories high, looking for victims.
Witnesses reported a strong smell of gas in the area. Gas utility Consolidated Edison said it shut off service to area buildings.
CBS News affiliate WCBS reports that neighborhood residents said they had complained to the city several times about the construction at the site, saying crews worked illegal hours and the building was going up too fast.
"I used to walk around the block because it was imminently dangerous. They were swinging buckets of concrete over passing cars on 51st Street, and the crane was not properly attached to the building it was constructing," Neighborhood resident Bartle Bull told CBS News correspondent Wendy Gillette.
City Building Department records showed that on March 4, a caller told officials that the upper portions of the crane appeared to lack the proper number of safety ties attaching it to the building.
But, Gillette reports, late today a New York City building commissioner told reporters that the crane had been inspected yesterday and it was considered safe.
Another call questioning the crane's safety was dismissed as unwarranted by another inspector in February.
Kaplan said the company had subcontracted the work to different companies and was not in charge of the crane. He wasn't sure if any workers at the site were among those killed.
Bill Reilly, a retired UPI reporter, said he was in his apartment a block away when it happened.
"First I heard a rumble, and it increased, and then it increased," Reilly said. "It continued building in strength until there was a final vroom! It shook the six story brick building that I live in."
Police blocked traffic in the area. Gawkers crowded the streets, snapping cell phone pictures and stopping to point at the wreckage. Fire trucks filled area streets and chunks of the building littered the ground.
The catastrophe comes amid a building boom in New York City and follows a spate of construction accidents in recent months, including some involving cranes.
At a Donald Trump hotel-condominium tower in SoHo, a worker plummeted 40 stories to his death last month when a concrete form gave way. A month before that, a crane's nylon sling broke away and dropped seven tons of steel onto a construction trailer across from ground zero, injuring an architect.