"I would say the agreement has taken a downward trend," said Sen. Bob Corker, R-Tenn., chairman of the Senate Foreign Relations Committee.
"If, in fact, it's as bad a deal as I think it is at the moment, we will do everything that we can to stop it," House Speaker John Boehner said.
And it wasn't just Republicans. Many Democrats expressed reservations, too.
"The deal doesn't provide for the anytime, anywhere inspections that we were all talking about," said Sen. Bob Menendez, D-New Jersey.
"The deal also lifts the arms embargo on conventional weapons in five years and potentially missile technology in eight or sooner," Menendez said. "Those are problematic because Iran is the biggest supporter of terrorism in the world."
In fact, one of the only people at the Capitol expressing strong support was former Secretary of State Hillary Clinton, who was on the Hill meeting with congressional Democrats.
"This is an important step in putting the lid on Iran's nuclear program," Clinton said.
But Republicans, like Senate Majority Leader Mitch McConnell, argue that lifting economic sanctions will only make Iran richer, stronger and more dangerous.
"The White House says that any deal when you are negotiating with your enemies is not going to be perfect. Do they have a point?" I asked McConnell.
"This is considerably short from perfection, apparently! We knew what was working, sanctions. Imagine what we might have now if we had spent the last two years ratcheting up the sanctions, as opposed to this."
McConnell acknowledged that even if Congress votes down the deal, the president will promptly issue a veto. And mustering a full two-thirds of Congress to overcome that veto will be, if not impossible, very challenging.