"That would be a mistake," he told a Senate panel. He said Americans just have to stay the course and ignore the naysayers.
"Timing in war is never predictable. There are never guarantees," Rumsfeld told the Senate Armed Services Committee. "Those who say we are losing this war are wrong. We are not."
Congressional Democrats are demanding answers about the future presence of U.S. troops in Iraq.
Rumsfeld, testifying on the progress in training Iraq's own security forces, said these forces have "a way to go," but progress was being made.
"Success will not be easy and it will require patience. ... But consider what has been accomplished in 12 months," Rumsfeld said, citing elections in January, economic improvements, and an increasingly improving security force.
The Bush administration contends that Iraqis must be able to defend their own country against a lethal insurgency before a timeline for bringing home troops can be considered.
But progress has been slower than expected. In recent weeks, insurgents have increasingly targeted Iraqi security forces. And U.S. casualties, war spending and public skepticism continue to climb, ruffling both Republicans and Democrats.
"Leaving before the task is complete would be catastrophic," Gen. Richard Myers, chairman of the Joint Chiefs of Staff, told the panel.