On Thursday, the furor came full circle when the Sons of Confederate Veterans turned on Allen.
The organization criticized the Republican for saying he had been slow to grasp the pain that Old South symbols like the Confederate flag cause black people.
"The denunciation of the flag to score political points is anathema to our organization," Brag Bowling, immediate past state commander of the SCV, told reporters near the state Capitol, once the Confederacy's seat of government.
Allen campaign manager Dick Wadhams said Allen stands by his comments.
What had been a warm-up race for a 2008 presidential bid has become the fight of Allen's political life. He had been a strong favorite over Democrat Jim Webb, a decorated combat veteran who once served as President Reagan's Navy secretary.
"Before the Macaca incident, this race was over. George Allen had a 16-point lead in the polls and had a huge financial advantage over his opponent, who had no money. In one moment, he took a race that was essentially won and turned it competitive overnight," said Robert D. Holsworth, a political science professor at Virginia Commonwealth University.
Since Aug. 11, when Allen singled out Webb campaign volunteer S.R. Sidarth at a mostly white rally and called him "macaca" — a term considered by some to be a racial slur — the senator has been beset by damaging claims from his past. Allen's campaign gave conflicting accounts of how he came up with the word.
College-era acquaintances then claimed he used a common racial slur for blacks. Stories about how the Confederate flag he kept in his home and a hangman's noose in his law office have gained new currency. At a debate, Allen was startled and irritated by a question about whether his grandfather was Jewish, then confirmed it a day later.
Allen now has even Republicans publicly questioning what has gone wrong with a candidate known for disciplined, surefooted campaigns in two convincing statewide triumphs the past 13 years.
"Allen's mistake is he's taken his re-election for granted and he's been getting his nice presidential team together," conservative commentator Bay Buchanan said last week on CNN.