"I think it's based on racism," Carter said in response to an audience question at a town hall held at his presidential center in Atlanta. "There is an inherent feeling among many in this country that an African-American should not be president."
The Georgia Democrat said the outburst was a part of a disturbing trend directed at the president that has included demonstrators equating Obama to Nazi leaders.
"Those kind of things are not just casual outcomes of a sincere debate on whether we should have a national program on health care," he said. "It's deeper than that."
Wilson, a South Carolina Republican, was formally rebuked Tuesday in a House vote for shouting "You lie!" during Obama's speech to Congress last Wednesday.
The shout came after the president commented that illegal aliens would be ineligible for federal subsidies to buy health insurance. Republicans expressed their disbelief with sounds of disapproval, punctuated by Wilson's outburst.
Tuesday's rebuke was a rare resolution of disapproval pushed through by Democrats who insisted that Wilson had violated basic rules of decorum and civility. Republicans characterized the measure as a witch hunt and Wilson, who had already apologized to Obama, insisted he owed the House no apology.
Wilson's spokesman was not immediately available for comment, but his eldest son defended his father.
"There is not a racist bone in my dad's body," said Alan Wilson, an Iraq veteran who is running for state attorney general. "He doesn't even laugh at distasteful jokes. I won't comment on former President Carter, because I don't know President Carter. But I know my dad, and it's just not in him."
"It's unfortunate people make that jump. People can disagree - and appropriately disagree - on issues of substance, but when they make the jump to race it's absolutely ludicrous. My brothers and I were raised by our parents to respect everyone regardless of background or race."
South Carolina's former Democratic Party chairman said that he doesn't believe Wilson was motivated by racism, but said the outburst encouraged racist views.
"I think Joe's conduct was asinine, but I think it would be asinine no matter what the color of the president," said Dick Harpootlian, who has known Wilson for decades. "I don't think Joe's outburst was caused by President Obama being African-American. I think it was caused by no filter being between his brain and his mouth."
Harpootlian said he received scores of racial e-mails from outside South Carolina after he talked about the vote on Fox News.
"You have a bunch of folks out there looking for some comfort in their racial issues. They have a problem with an African-American president," he said. "But was he motivated by that? I don't think so. I respectfully disagree with President Carter, though it gives validity to racism."
Carter called Wilson's comment "dastardly" and an aftershock of racist views that have permeated American politics for decades.
"The president is not only the head of government, he is the head of state," he said. "And no matter who he is or how much we disagree with his policies, the president should be treated with respect."