"Saddam Hussein is now taking his rightful place alongside Hitler, Stalin, Lenin, Ceausescu in the pantheon of failed brutal dictators and the Iraqi people are well on their way to freedom," Rumsfeld told a Pentagon briefing.
"This is a very good day," Rumsfeld said as U.S. forces stormed through the streets of the Iraqi capital and were greeted by jubilant Baghdad residents.
But fighting continued in parts of the city and Saddam's whereabouts remained uncertain.
In a square in central Baghdad, U.S. Marines helped a crowd of Iraqis topple a giant statue of Saddam in a bold symbolic gesture.
CBS News Analyst Col. (Ret.) Mitch Mitchell called the event a "psychological victory."
"Bringing it down is symbolic of the fall of the regime — that the regime no longer is in control of the country," says Mitchell. "It is a great psychological victory to topple the statue of Saddam that was erected to himself in 2001.
"This is not the end of the war, but it is very close to it."
In other developments:
Rumsfeld cautioned that despite the day's developments, there still remained work to do and hard fighting.
He said U.S.-led forces still needed to find the seven American prisoners of war who were captured by Iraqi troops and locate the regime's weapons of mass destruction. He encouraged Iraqi scientists, military officers and others to come forward with information, saying "rewards are available to those who help us."
Rumsfeld also said Syria was still serving as a conduit for military equipment, including night-vision goggles, heading to Iraqi forces. He also suggested that the Syrian government was helping Saddam loyalists to leave Iraq and cross the border into Syria.
With the regime's feared security forces nowhere to be seen, Iraqis dared to cheer U.S. troops and attack the symbols of Saddam's rule.
They danced in the streets, waving rifles, palm fronds and flags, and defaced posters of the longtime Iraqi president — acts that would have been unthinkable days or weeks before.
"I'm 49, but I never lived a single day. Only now will I start living," said Yussuf Abed Kazim, a preacher at a local mosque who bashed the Saddam statue with a sledgehammer as other Iraqis yelled, "Hit the eye! Hit the eye!"
"That Saddam Hussein is a murderer and a criminal," Kazim said.
Elsewhere, U.S. troops continued to encounter pockets of resistance.
No more than a mile and a half away from central Baghdad, U.S. Marines came under fire near the Ministry of Oil building, CBS News Correspondent Byron Pitts reports.
"In the open desert, in the street, the Iraqis have proven to be an unworthy adversary to the U.S. military," reports Pitts. "But in an urban setting, hiding in doorways and windows, they have proven quite capable to cause damage."
The street celebrations in Baghdad, especially in Shiite Muslim neighborhoods, erupted after fighting subsided overnight.
Iraqis were already looting government ministries, police stations, universities and even the headquarters of the Iraq Olympic Committee, making off with computers, furniture, even military jeeps. One young man used roller skates to wheel away a refrigerator.
In the north, celebrations also broke out in at least two cities in the Kurdish autonomous region. Some honked their horns, others chanted "George Bush! George Bush!"
Some Iraqis were not rejoicing Wednesday.
"This is the destruction of Islam," said Qassim al-Shamari, 50, a laborer wearing an Arab robe. "After all, Iraq is our country. And what about all the women and children who died in the bombing?"
State television was off the air and foreign journalists said their "minders" – government agents who monitor their reporting – did not turn up for work Wednesday.
There was no sign of Iraqi Information Minister Mohammed Saeed al-Sahhaf, whose daily briefings had constituted the main public face of the regime during the war.
"It's a historic moment," White House spokesman Ari Fleischer said of the scenes of jubilation in the streets of Baghdad.
He said it reflects the desire of Iraqis to be free. But he cautioned that "we are still in the middle of a shooting war" and there are sections of Baghdad and other Iraqi cities in which coalition forces face a threat.
British Prime Minister Tony Blair was also cautious. "This conflict is not over yet," he told the House of Commons. "There are still some very difficult things to do. As we speak, there is still intense resistance ... among those parts of Saddam's regime that want to cling onto power."
Speaking to the annual convention of the American Society of Newspaper Editors, Vice President Dick Cheney said the advance of U.S. troops into Baghdad is proof that early criticism of the war's plans was misguided.
"With every day, with every advance of our coalition forces, the wisdom of that plan becomes more apparent," Cheney said.