Lieberman, who is seeking a fourth term as an independent candidate after losing the Democratic nomination to newcomer Ned Lamont on Aug. 8, also criticized some fellow Democrats. He said several Democrats are trying to impose a "litmus test" on the party.
"I think it's still time for new leadership at the Pentagon," he said on CBS's Face the Nation.
Lieberman, an early supporter of the Iraq war, said he called for Rumsfeld to step down in 2003.
"With all respect to Don Rumsfeld, who has done a grueling job for six years, we would benefit from new leadership to work with our military in Iraq," he told Bob Schieffer.
Lieberman said the Bush administration should have sent more troops into Iraq "to secure the country."
"We had a naive vision that the Iraqis were going to embrace us and then go on and live happily ever after," he said.
Lieberman also said the administration must "put severe pressure on the Iraqis to contain sectarian violence."
"There is still hope in Iraq and as long as there is we cannot just pick up and walk away and leave them to the sure disaster that would follow and would compromise our security in the war on terrorism," he said.
Lieberman said he would support an "international crisis conference on Iraq" with the United States, its allies and Arab countries worrying "that if Iraq collapses and falls into civil war that Iran will surge in and dominate and claim a victory."
He also accused Lamont of distorting his stance on Iraq.
"He made me into a cheerleader for George Bush and everything that's happened," Lieberman said. "And the record shows that, while I believe we did the right thing in overthrowing Saddam Hussein, I've been very critical over the years, particularly in 2003 and 2004, about the failure to send enough American troops to secure the country, about the absence of adequate plans and preparation to deal with post-Saddam Iraq."
"As bad as things are now — and they've gotten worse in the last six months — it would be a disaster if America set a deadline and said we're getting all of our troops out by a given date," Lieberman said. "That's a position Ned Lamont has taken."
Responding to a question about Sen. John Kerry, D-Mass., who was quoted as saying that Lieberman echoes Republicans, Lieberman said it was "just plain politics by somebody who has ambitions of his own."
"I voted 90 percent of the time with a majority of Democrats in the U.S. Senate," he said.
"I'm worried that my party may become what we've accused the Republicans of, a kind of litmus-test party," he said. "If you don't agree with us 100 percent of the time, you don't agree with us. I'm devoted to the Democratic Party."