House Democratic Whip Steny Hoyer on Thursday took issue with House Minority Leader Nancy Pelosi's comments that "five white guys" -- a group that includes Hoyer -- are the ones working on a fix for Deferred Action for Childhood Arrivals (DACA) recipients.

"The 'five white guys' I call them, you know," Pelosi said, referring to the group of five bipartisan lawyers, in a press conference Thursday.

"Are they going to open a hamburger stand next or what?" Pelosi said, a reference to the popular Five Guys fast-food burger joint. Pelosi argued the lack of any minority involvement was not helpful in the negotiations. White House chief of staff John Kelly, House Democratic Whip Steny Hoyer, House Republican Whip Kevin McCarthy, Senate Democratic Whip Dick Durbin and Senate Republican Whip John Cornyn are looking to hammer out details of an immigration deal.

Hoyer told CBS News he took offense to Pelosi's offhand comment as he works to find a solution for DREAMers.

"That comment is offensive. I am committed to ensuring DREAMers are protected and I will welcome everyone to the table who wants to get this done," Hoyer said.

Pelosi's office did not back away from her comments later Thursday.

"Leader Pelosi has every confidence in the leadership of Whip Hoyer and Sen. Durbin, which they have demonstrated on this topic for years... It's not a question of who's there, but who's not there," Pelosi spokesman Drew Hammill told CBS News.

A bipartisan group of senators announced Thursday they had reached an "agreement in principle" on immigration and DACA, although the details haven't been publicly released.

Republicans also criticized Pelosi Thursday for calling some companies' $1,000 bonuses for workers -- which they say are the result of tax reform -- "crumbs."

Ronna McDaniel, chairwoman of the Republican National Committee, derided the comment on Twitter, along with others in the GOP.

CBS News' Alicia Amling and Nancy Cordes contributed to this report.