His back-to-back scandals this year - a crude joke about Sarah Palin's daughter and revelations through an alleged blackmail attempt by a CBS producer that Letterman had sexual relationships with "Late Show" staff members - are likely to hit his reputation hardest with women.
There are still many unknowns following Letterman's startling admission last week: How many women were involved? Did the relationships end well? Did women feel pressured by the boss' advances, or were they happily consensual? Did any other staffers feel the way to get ahead was through Letterman's bed? Are women who did not have intimate relations with Letterman upset that those who did may have gotten special treatment or career advances? Is anyone talking to lawyers?
Watch Letterman's Remarks
Photos: David Letterman
All could determine whether Letterman is significantly damaged or if it's just a footnote to his career.
Certainly, it can't help when the New York Post blares on its front page: "Dirty Dave's Harem."
"It's very interesting. Men and women think very differently about this story," NBC News anchor Ann Curry said. "The men I've talked to think, `How could a person within your own company kind of betray you like this?' But the reaction I'm hearing from women is completely on the other end. They're saying, `How could you have affairs, multiple affairs, with members of your own staff and how does that create a fair and equal working environment?'"
Names Surface of Women Linked to Letterman
Will "Other Shoes" Drop for Letterman
Viewers Split on Letterman Revelations
Experts: Letterman Was Right To Come Clean
Halderman's Lawyer: Don't Rush To Judgment
Halderman Pleads Not Guilty to Blackmail
Many women who work in television, and probably most other industries, can recall feeling pressured to do something with a man she works for, said Curry's colleague, Kathie Lee Gifford.
"I'm not saying David did that, but the details are yet to be discovered," she said. "They might have been very willing partners who initiated it. We don't know, but the minute we hear stories like that we go, `Uh-huh.'"
Letterman's prickly personality and sarcastic humor seem tailor-made for a young, male audience. But the facts tell a different story: Letterman's typical audience was 58 percent female last season, with an average age of just under 55, according to the Nielsen Co.
In other words, he should care very much whether women are disgusted by his behavior and reject his comedy as a result.
Letterman effectively seized control of his story the first night by establishing himself as a crime victim. The way he revealed his transgressions _ by talking about the "terrible, terrible" and "creepy" things that he did before saying what they were _ left enough dark thoughts running through viewers' minds that sex with consenting age staff members seemed mild as a result.
His television performance was judged more than his conduct. The Associated Press said in its headline that Letterman had created a "brilliant, unsettling hour of TV." Even Linda Stasi of the New York Post, whose gleeful headline writers couldn't resist "Laid Nights With David Letterman," said the talk show host's "explanation-as-monologue was nothing short of brilliant."
Many of Letterman's fans seem to recognize he's an oddball in real life, a man who radiates unapproachability, keeps his studio at freezer temperatures and is prone to constant brooding about past failures (Jay Leno has already left the "Tonight" show, but Letterman still can't get over losing that gig to him in 1993). Sexual affairs? Seems normal by comparison.
While Letterman's representatives have released few details about the relationships, they have said they predate his marriage to Regina Lasko in March. The couple has a 5-year-old son.
"I don't really care who someone sleeps with as long as it's not coerced and as long as there's not some explicit or implicit promise of favors or the like," Kim Gandy, former president of the National Organization for Women, said. "It's another adult _ it's not a minor. If that's all it is, he's a single guy and he had a fling."
But Gene Grabowski, an expert in crisis public relations for Levick Strategic Communications, said that after a couple of days, the extortion attempt will recede and viewers will focus on Letterman's behavior. Letterman said on Thursday that he doesn't plan to address the issue again. Grabowski said he needs to.
"People want to forgive David Letterman," he said. "People like David Letterman. But he has to do his part to make people want to support him, and that means apologize."
After Letterman's joke about Palin's daughter being "knocked up" by Alex Rodriguez fell horribly flat, he gave a relatively cursory explanation and called it a failed attempt at comedy. After a weekend's reflection, he came back to give a much stronger apology.
The latest mess comes at a time when Letterman is re-establishing himself as the dominant figure in late-night television. His "Late Show" is nearly must-see TV. Within the past two weeks, he has featured President Barack Obama, Madonna and a riveting personal scandal. The best Conan O'Brien could do was a concussion when a stunt went awry.
Yet the women in Letterman's audience may need some reassuring.
Associated Press Television Writer Lynn Elber in Los Angeles and AP Writer Mesfin Fekadu in New York contributed to this report.
By Dave Bauder