The ousted official denies the allegation and so does the farmer, CBS News Chief Legal Correspondent Jan Crawford reports.
The allegations started with a speech by an Agriculture Department official describing her attitude 24 years ago toward a white farmer.
"Here I was, faced with having to help a white person save their land, so I didn't give him the full force of what I could do," the official, Shirley Sherrod, said.
That comment, in a speech to the National Association for the Advancement of Colored People, lit up the blogosphere after a conservative website this week aired it and suggested there was reverse racism in the administration.
Vilsack reacted swiftly. He said Tuesday the USDA has "zero tolerance for discrimination" and fired Sherrod for those comments.
"When I saw the statements and the context of the statements, I determined that it would make it difficult for her to do her job as a rural development director," Vilsack said.
But, as with so many issues of race, there is a lot more to this story.
More on Shirley Sherrod
Sherrod said later in the same speech she was wrong and ultimately helped the man save his farm but that didn't air on the Internet.
And the farmer and his wife have defended her.
"She was just as nice as she could be," said Roger Spooner. "As far as racism, I think that's somebody who wants to start something."
Sherrod has angrily answered charges of racism. She told CNN she was unfairly forced out by a White House skittish about issues of race.
"I had at least three calls telling me the White House wanted me to resign," Sherrod said.
Vilsack said the decision was his alone.
"I made this decision; it's my decision," said Vilsack. "Nobody from the White House contacted me about this at all."
But in this growing controversy, this much is clear: Shirley Sherrod - now out of a job - helped Roger and Eloise Spooner.
"She saved our farm, 400 something acres, almost 500 acres," said Roger Spooner. "She saved our farm."
Although the NAACP initially condemned Sherrod's remarks, the organization released a statement Tuesday evening saying it . Now they're completely changing course and are urging Vilsack to reconsider firing Sherrod. The organization said Sherrod's speech was deliberately edited to create a false impression of racial bias.