The portrait of former House Speaker Dennis Hastert, R-Illinois, was removed from a hallway outside of the House chamber on Monday.
Newly-elected Speaker Paul Ryan made the decision to remove the portrait from the Speaker's lobby just days after he took the gavel and after Hastert pleaded guilty last week to a felony charge of breaking federal banking laws.
"The Speaker believed it was appropriate to rotate in a different portrait," Ryan spokeswoman AshLee Strong told CBS News.
In its place is a portrait of Frederick Gillet, a Republican who represented Massachusetts and who served as speaker of the House from 1919 to 1925.
Groups had been calling for the removal of Hastert's portrait for months after allegations of sexual abuse arose at the same time that the federal scandal unfolded.
According to the federal indictment, Hastert paid $3.5 million to an unidentified person to cover up past wrongdoing. Law enforcement officials told CBS News that the money was used to cover up claims of sexual misconduct from Hastert's days coaching a high school wrestling team in Illinois.
A network for sexual abuse survivors, Survivors Network of those Abused by Priests, demanded in June that it be taken down.
The removal of Hastert's portrait is unprecedented. It hung on the walls outside of the House chamber since 2009 alongside other portraits of former speakers.
Hastert, 73, served as speaker from 1999 until 2007 and served in Congress since 1987. His alma mater, Wheaton College, removed his name from its Center for Economics, Government and Public Policy earlier this year.
CBS' Walt Cronkite contributed to this story.