Word comes a day after big Democratic gains in the midterm elections, in which Rumsfeld was a focus of much of the criticism of the Iraq war.
, former head of the CIA, will replace Rumsfeld, Mr. Bush said at a White House news conference.
The president lavished praise on Rumsfeld, who has spent six stormy years at his post.
"Don Rumsfeld was a superb leader during a time of change, and he appreciates a fresh perspective in a critical period in this war," Mr. Bush said. He called Rumsfeld a "trusted adviser and friend," and said he's "deeply grateful" for his service to the country.
"I recognize that many Americans voted last night to register their displeasure with the lack of progress being made" in Iraq, the president said. "Yet I also believe most Americans – and leaders here in Washington from both political parties – understand we cannot accept defeat."
Asked whether there would be a new direction in a war that has claimed the lives of more than 2,800 U.S. troops and thousands of Iraqis, Mr. Bush replied, "Well, there's certainly going to be new leadership at the Pentagon."
He disclosed he had spoken with Gates last Sunday, two days before congressional elections in which Democrats swept to control of the House of Representatives and possibly the Senate.
"Bob Gates will bring a fresh perspective and great managerial experience," Mr. Bush said.
Democrats have long called for Rumsfeld's resignation, reports CBS News correspondent Bob Fuss, but the president just before the election said he supports Rumsfeld and the way the war in Iraq was conducted and wanted him to stay.
Not surprisingly, Democrats reacted positively to the news of Rumsfeld's departure.
"I welcome the long overdue change in leadership at the Pentagon," House Democratic leader Nancy Pelosi, D-Calif., said in a statement, "now we need a change in policy."
Rumsfeld, 74, has served as Mr. Bush's defense secretary since the beginning of his first term in 2000. It was his second tour of duty as defense chief. He first held the job a generation ago, when he was appointed by President Ford.
Gates, 63, is the president of Texas A&M University and a close friend of the Bush family. He served as CIA director for Mr. Bush's father from 1991 until 1993.
Gates first joined the CIA in 1966 and served in the intelligence community for more than a quarter century, under six presidents.
His nomination must be confirmed by the Senate.