Speaking Wednesday on ABC's "Good Morning America," Gingrich said Pelosi should not have to leave Congress but should not be Speaker.
"What she said Thursday is a stunning, dishonest statement," he said. "She really disqualified herself to be speaker."
Gingrich repeated that message in a letter published Wednesday in the conservative periodical Human Events. In the letter, titled "Why Pelosi Should Step Down," he says Pelosi has damaged America's safety.
"The case against Nancy Pelosi remaining Speaker of the House is as simple as it is devastating," he wrote. "The person who is No. 2 in line to be commander in chief can't have contempt for the men and women who protect our nation."
Last Thursday, Pelosi said the CIA in a fall 2002 briefing misleadingly told her that detainees in the war on terror had not been subject to waterboarding. She put out a statement Friday saying that her comments were directed toward the Bush administration, wich authorized the CIA's use of waterboarding -- not "the dedicated men and women of the intelligence community...who work to keep our country safe."
In his letter, Gingrich contends that Pelosi was, in fact, informed about the use of waterboarding.
"To avoid culpability for the choices she supported, she's now telling us she didn't know," he wrote. "And she's calling the intelligence officials who say otherwise liars and criminals."
While Democrats have tried to play down the controversy surrounding Pelosi's allegation against the CIA, Gingrich and other Republicans continue to express their disapproval of the Pelosi's claims.
Other Republicans have been critical of the speaker as well. On Sunday, House Minority Leader John Boehner (R-Ohio said she should either apologize for her allegation or provide proof she was lied to. The Politico reports Boehner has continued to criticize the speaker on the issue in television interviews, press conferences and newspaper op-eds.
Democrats, however, have supported Pelosi's claims.
"I believe the speaker when she says she was not briefed," House Majority Leader Steny Hoyer (D-Md.) said Tuesday, according to the Hill newspaper.
President Obama has also shown his support for Pelosi. During a Tuesday morning announcement he made regarding fuel efficiency standards, the president said he wanted to thank Pelosi "who has just been cracking the whip and, you know, making Congress so productive over these last several days."
"We are grateful for her," he said.