The comment, which the president made in an afternoon taping of CBS' "The Late Show," prompted laughter from the audience and this response from host David Letterman: "How long have you been a black man?"
(Watch more clips from Letterman's interview with President Obama at the bottom of this post)
Mr. Obama said the notion that racism is playing a role in the criticism, which has been voiced by former President Jimmy Carter and others, is countered in part by the fact that he was elected in the first place – which, he said, "tells you a lot about where the country's at."
"One of the things that you sign up for in politics is that folks yell at you," the president said, noting that "whenever a president tries to bring about significant changes, particularly during times of economic unease, there is a certain segment of the population that gets very riled up." He pointed to the experiences of Franklin Delano Roosevelt, John F. Kennedy and Ronald Reagan as examples.
The Letterman interview, which comes on the heels of his appearance on five different networks on Sunday, marks the latest public relations effort from a president trying to win public support for his health care efforts. Perhaps in honor of the forum, Mr. Obama opened with a joking reference to Letterman's mention that an audience member had brought a heart-shaped potato to the taping.
The main reason he had come on the show, Mr. Obama quipped, was "to see that heart-shaped potato." After Letterman retrieved the potato, the smiling president deadpanned, "That's remarkable. This is remarkable." He then, with the audience member's consent, kept the potato.
The appearance was not all jokes; Mr. Obama said that the economy was improving but that employment was lagging behind.
"Unemployment is still going to be a big problem for at least another year," he said, though he insisted the economy would emerge "stronger than before."
On Afghanistan, the president said that he will not make a decision whether to send more troops until he decides on a strategy following a comprehensive review. The top commander in Iraq has warned that more troops are needed for the U.S. to have a chance to emerge victorious.
Asked by Letterman about the wisdom of the war in Iraq, Mr. Obama said, "because Saddam Hussein is not there, that's a good thing. He was somebody who certainly had aspirations to cause a lot of trouble."
Mr. Obama added, however, that "that given the enormous stakes we had in Afghanistan, we should have finished the job there."
The president, who appeared to a standing ovation, told Letterman that his kids had been doing well, and that "they basically just goofed off all summer, which I couldn't do." He said their lives had been relatively normal and that "people don't really give them too much of a fuss" when they go out.
Mr. Obama has appeared on the "Late Show" five times before, though never as president. The interview will air in full Monday evening.