Gregg, a fiscal conservative, said differences over the stimulus package and the way in which the U.S. Census will be run led to his decision.
"I've been my own person for 30 years," Gregg said at a press conference explaining his withdrawal. "It's hard to be part of the team and not be 110 percent a part of the team."
President Obama said Thursday afternoon that Gregg's decision came as "something of a surprise" to him. The president later added that Gregg's withdrawal won't deter him from working with the GOP or trying to change partisan ways of Washington.
In an earlier statement, Gregg said that he and the Obama administration "are functioning from a different set of views on many critical items of policy." He said at the press conference that he erred in not considering the implications of the policy differences earlier.
"I said yes," Gregg said. "That was my mistake."
An adviser close to the White House tells CBS News that the White House is "stunned" and "angry" about the decision.
A Democratic source, meanwhile, told CBS News that Gregg "sat with the president, said he wanted the job, knew his policies and erratically dropped out without warning," said the source.
In a statement, White House press secretary Robert Gibbs said Gregg had been "very clear throughout the interviewing process that despite past disagreements about policies, he would support, embrace, and move forward with the President's agenda."
"Once it became clear after his nomination that Senator Gregg was not going to be supporting some of President Obama's key economic priorities, it became necessary for Senator Gregg and the Obama administration to part ways," Gibbs said. "We regret that he has had a change of heart."
A White House official tells CBS News that Gregg first called White House Chief of Staff Rahm Emanuel on Monday to convey his intentions - and met with President Obama Wednesday.
Gregg, a 61-year-old Republican, had been President Obama's pick to become Commerce Secretary after his first choice, New Mexico Gov. Bill Richardson, withdrew from consideration amid a federal investigation into possible pay-to-play deals for contracts in his state.
"I couldn't be Judd Gregg and serve in the Cabinet. I should have faced up to the reality of that earlier," Gregg told Politico this afternoon. "I've been my own person and I began to wonder if I could be an effective team player. The president deserves someone who can block for his policies. As a practical matter I can contribute to his agenda better — where we agree — as a senator and I hope to do that."
"The fault lies with me," he added. "I may have embarrassed myself but hopefully not him."
After concerns were expressed about Gregg running the census in his capacity as Commerce Secretary, the White House suggested it would play a larger role in performing the population count. That angered Republicans, who accused the Obama administration of meddling in the count for political gain.
Congressional districts are drawn on the basis of census results, and they also influence how much federal funding goes to particular areas.
"If President Obama doesn't trust Sen. Gregg to oversee a fair and accurate census, he should withdraw the nomination," GOP conference chairman Mike Pence said.
Gregg has served in the Senate since 1993, and previously was New Hampshire governor and a member of the House of Representatives.
At his press conference, Gregg had kind words for President Obama, saying he has been "incredibly gracious."
"I immensely respect him, and I know he's going to be a strong, and effective and good president," Gregg said. He added that there were issues on which he could ""carry his water here in the Senate."
Gregg is the latest Obama administration nominee to withdraw from consideration. Last week, both former Senate Majority Leader Tom Daschle and Nancy Killefer withdrew their nominations to become health and human services secretary and chief performance czar over tax issues.
"I realize that to withdraw at this point is really unfair in many ways, but to go forward and take this position…would have been an even bigger mistake," Gregg said. "The bottom line is this was simply a bridge too far for me."