But for others, as CBS News Correspondent Richard Schlesinger reports, the flag is more complex issue than that.
Southerners who see this flag remember their ancestors who served under it.
"He went. He served gallantly and Im proud of him," says Kathryn Boone Hamilton of her great grandfather Jason Boone.
Its not the words that are surprising; its the source. Hamilton, who is black, has spent years studying the life of her great grandfather, who is also black, and now he has a new headstone with his unit name on it: Company K 41st Virginia Infantry, Confederate States of America.
A handful of black people have discovered a connection to the Confederacy. Last year, when Boones great granddaughter found out about his Confederate service, she asked for this memorial ceremony complete with the Confederate flag.
"Growing up, I did think of it as a racist symbol, Hamilton says. "But after finding out that Jason fought for the Confederacy, I look at them differently now."
Katherine Hamilton knows this flag offends most black people and doesnt support flying it on public buildings, even though shes accepted it as part of her private history.
"Jason fought under this flag," she says. "Am I expected to say I regret he did it or feel guilty about it?
"Thats not a white mans flag. Thats not a black mans flag. Its our flag, says Robert Harrison, a research librarian at South Carolina State University. He believes one of his ancestors might have fought for the Confederacy.
"Im a defender of history and its my point to make sure that all of it gets told," he says.
Its hard to believe, but historians agree, at least several hundred blacks joined Confederate soldiers during the war. But thats all they agree on. How many there were and whether blacks actually fought or merely acted as servants or laborers may never really be known.
"There is some evidence that, in the heat of battle, occasionally they would take up arms, especially if their owner, their master happened to be killed or wounded," says TK McPherson, a Pulitzer Prize-winning historian who says the few black people who feel an attachment to the rebels should worry about being used by people who want to ignore the legacy of slavery.
"If they can trot out a black spokesperson to say, yes my ancestor fought for the Confederacy and Im proud that he did, that helps give them legitimacy." he says.
Kathryn Boone Hamilton disagrees. "In a way, thats insulting to me," she says. "Beause its saying that I dont have the common sense to recognize an ulterior motive."
People like Kathryn Hamilton and Robert Harrison find themselves in historys gray area, black people linked to a white mans symbol, by their own heritage.