In a speech to Justice Department employees marking Black History Month, Holder said the workplace is largely integrated but Americans still self-segregate on the weekends and in their private lives.
"Though this nation has proudly thought of itself as an ethnic melting pot, in things racial we have always been and I believe continue to be, in too many ways, essentially a nation of cowards," Holder said.
Race issues continue to be a topic of political discussion, but "we, as average Americans, simply do not talk enough with each other about race."
Holder's speech echoed President Barack Obama's landmark address last year on race relations during the hotly contested Democratic primaries, when the then-candidate urged the nation to break "a racial stalemate we've been stuck in for years" and bemoaned the "chasm of misunderstanding that exists between the races." Obama delivered the speech to try to distance himself from the angry rhetoric of his former pastor, the Rev. Jeremiah Wright.
Holder cited that speech by Obama as part of the motivation for his words Wednesday, saying Americans need to overcome an ingrained inhibition against talking about race.
"If we're going to ever make progress, we're going to have to have the guts, we have to have the determination, to be honest with each other. It also means we have to be able to accept criticism where that is justified," Holder told reporters after the speech.
Holder urged people of all races to use Black History Month as a chance for honest discussion of racial matters, including issues of health care, education and economic disparities.
Race, Holder said, "is an issue we have never been at ease with and, given our nation's history, this is in some ways understandable... If we are to make progress in this area, we must feel comfortable enough with one another and tolerant enough of each other to have frank conversations about the racial matters that continue to divide us."
In a country founded by slave owners, race has bedeviled the nation throughout its history, with blacks denied the right to vote just a few decades ago. Obama's triumph last November as well as the nomination of Holder stand as historic achievements of two black Americans.
Holder told hundreds of Justice Department employees gathered for the event that they have a special responsibility to advance racial understanding.
Even when people mix at the workplace or afterwork social events, Holder argued, many Americans in their free time are still segregated inside what he called "race-protected cocoons."
"Saturdays and Sundays, America in the year 2009 does not in some ways differ significantly from the country that existed almost 50 years ago. This is truly sad," said Holder.
Andrew Grant-Thomas, Deputy Director of the Kirwan Institute for the Study of Race and Ethnicity at Ohio State University, praised Holder's general message but said the wording of the speech may alienate some.
"He's right on the substance, but that's probably not the most politic way of saying it. I'm certain there are people who will hear him and say, 'That's obnoxious,"' he said, adding that what was missing from Holder's speech were specific examples of what painful subjects need to be addressed.
Hilary Shelton, vice president of the National Association for the Advancement of Colored People, called the speech "constructively provocative."
"Nobody wants to be considered a coward. We've learned to get along by exclusion and silence. We need to talk about it. People need to feel comfortable saying the wrong things," said Shelton.
Holder is headed to Guantanamo Bay early next week to inspect the terrorist detention facility there. Obama has assigned Holder to lead a special task force aimed at closing the site within a year.
Holder's Justice Department will have to decide which suspects to bring to U.S. courts for trial, which to prosecute through the military justice system, and which to send back to their home countries.