Rumsfeld also all but admitted that the U.S. military is losing the battle of ideas in Muslim world, reports CBS News Pentagon correspondent David Martin.
"He's just flat wrong," Rumsfeld said in response to a question about the controversial prison during an appearance at the Council on Foreign Relations. "We shouldn't close Guantanamo. We have several hundred terrorists, bad people, people that if let back out on the field would try to kill Americans. That's just a fact."
He said closing it would amount to pretending there is no problem with a terrorist threat to U.S. interests.
Rumsfeld took a swipe at Annan, saying, "He's never been to Guantanamo Bay," whereas representatives of the International Committee of the Red Cross "stayed there, lived there 24 hours a day" to observe conditions.
"That place is being run as well as any detention facility can be run," he added, his voice rising. "It's absolutely beyond comprehension," he said, that calls for closing Guantanamo Bay can be based on allegations of mistreatment and torture by the prisoners, whom Rumsfeld said are trained to lie.
A U.N. report issued earlier this week said some of the treatment of prisoners amounted to torture.
There are about 500 prisoners at Guantanamo Bay. Many have been held for nearly four years. Most were captured on the battlefields of Afghanistan in late 2001 after U.S. forces invaded in response to the terrorist attacks of Sept. 11, 2001.
Rumsfeld asserted that U.S. forces in Iraq are making progress on the security front, but he said there inevitably will be setbacks as the Iraqis struggle to assume control of their country.
"Our goal has to be to reduce our forces down, and to do it at a pace where we recognized that we're going to, I almost said, 'make a mistake.' It will look like a mistake. It's a judgment call. We're going to have to pull out of some pieces of real estate and turn over things to Iraqis, and they're going to drop the ball. Let's face it.