Lawmakers exiting a GOP meeting Friday said Ryan told them that Conroy was forced after lawmakers complained that had not been adequately tending to the pastoral needs of lawmakers.
"He had a number of complaints that the chaplain was not meeting the pastoral needs of the members in general," said Rep. Mike Coffman, R-Colo.
Ryan's office won't say why Conroy was asked to resign. It would be the first forced ouster in the history of the House.
While Ryan has the support of most of the chamber's Republicans, many Democrats are upset about Conroy's ouster and say politics may be behind it.
Democrats — and Conroy himself — have cited a prayer he offered last fall that called for fairness as the House debated tax cuts as a reason for GOP discontent with his performance.
Last November, Conroy prayed for lawmakers to make sure that "there are not winners and losers under new tax laws, but benefits balanced and shared by all Americans."
Conroy told the New York Times that shortly afterward Ryan warned him to "stay out of politics." Ryan himself is leaving office, though of his own volition, after 2018.
"He was essentially dismissed for praying and for this very gray and hazy representation that he was not ministering to some of the members," said Rep. Emanuel Cleaver, D-Mo., himself a clergyman. "Fairness and justice is an inseparable part of the Judeo-Christian theology."
One prominent Republican was upset as well.
"I'm not aware of any discontent or any criticism and to be the first House chaplain to be removed in the history of the Congress in the middle of a term raises serious questions. We deserve more of an explanation and why. Was there political pressure?" said Rep. Peter King, R-N.Y. "The speaker said it was just because certain people felt he was not complying with their requests or was not giving them good counseling. I never heard that from anyone."
Conroy has served as the chamber's chaplain since 2011 but offered his resignation last week at Ryan's urging. He will leave his House service next month.
Democrats on Friday sought to create a special committee to investigate "the motivations and actions" behind Ryan's decision to fire Conroy but the move was killed by a party-line vote.
Conroy's resignation letter said he was offering to step down at Ryan's request, calling his seven years of House service "one of the great privileges of my life."
The chaplain is responsible for opening the House each day with a prayer and offering counseling to lawmakers and aides on the House side of the Capitol.
Conroy is a Roman Catholic priest from the Jesuit order.