Rumsfeld also all but admitted that the U.S. military is losing the battle of ideas in Muslim world, reports CBS News Pentagon correspondent David Martin.
"He's just flat wrong," Rumsfeld said in response to a question about the controversial prison during an appearance at the Council on Foreign Relations. "We shouldn't close Guantanamo. We have several hundred terrorists, bad people, people that if let back out on the field would try to kill Americans. That's just a fact."
He said closing it would amount to pretending there is no problem with a terrorist threat to U.S. interests.
Rumsfeld took a swipe at Annan, saying, "He's never been to Guantanamo Bay," whereas representatives of the International Committee of the Red Cross "stayed there, lived there 24 hours a day" to observe conditions.
"That place is being run as well as any detention facility can be run," he added, his voice rising. "It's absolutely beyond comprehension," he said, that calls for closing Guantanamo Bay can be based on allegations of mistreatment and torture by the prisoners, whom Rumsfeld said are trained to lie.
A U.N. report issued earlier this week said some of the treatment of prisoners amounted to torture.
There are about 500 prisoners at Guantanamo Bay. Many have been held for nearly four years. Most were captured on the battlefields of Afghanistan in late 2001 after U.S. forces invaded in response to the terrorist attacks of Sept. 11, 2001.
Rumsfeld asserted that U.S. forces in Iraq are making progress on the security front, but he said there inevitably will be setbacks as the Iraqis struggle to assume control of their country.
"Our goal has to be to reduce our forces down, and to do it at a pace where we recognized that we're going to, I almost said, 'make a mistake.' It will look like a mistake. It's a judgment call. We're going to have to pull out of some pieces of real estate and turn over things to Iraqis, and they're going to drop the ball. Let's face it.
"And we're going to have to step in, go back in and fix it and then turn it back over again, and it's going to be three steps forward and one step back. It isn't going to be perfect, it isn't going to be pretty," he said.
Rumsfeld said U.S. forces already have closed or turned over to the Iraqis 30 military bases. He offered no projections of future U.S. troop withdrawals. Last December the Pentagon said the number of U.S. troops in Iraq would drop from about 138,000 to about 130,000 by March.
Rumsfeld also said al Qaeda and other Islamic extremist groups have poisoned the Muslim public's view of the United States through deft use of the Internet and other modern communications methods that the American government has failed to master.
"Our enemies have skillfully adapted to fighting wars in today's media age, but for the most part we, our country, our government, has not adapted," he said.
He quoted Ayman al-Zawahri, the chief lieutenant of al Qaeda leader Osama bin Laden, as saying that their terrorist network is in a media battle for the hearts and minds of Muslims. Rumsfeld agreed, saying that the battle for public opinion is at least as important as the battles on the ground in Iraq and Afghanistan.
The extremist groups are able to act quickly on the information front, with relatively few people, while the U.S. government bureaucracy has yet to keep up in an age of e-mail, blogs and instant messaging, he said.
"We in the government have barely even begun to compete in reaching their audiences," Rumsfeld said.
Rumsfeld wants the military to get better at putting out propaganda, which he admits is a touchy subject since it sometimes involves secretly planting good-news story overseas. As a model for future operations, Rumsfeld cited receipt military aid to the victims of the Pakistani earthquake, Martin reports.
"In some cases, military public affairs officials have had little communications training and little, if any, grounding in the importance of timing and rapid response, and the realities of digital and broadcast media," Rumsfeld said.