Well, actually, it was with women. But the founding editor of Playboy magazine also fell in love early with jazz, and both affairs have lasted a lifetime.
"It's the music of my youth. It's the music I grew up with. It's the music of my dreams," Hefner, looking wistful, said as he stood in the backyard of his Los Angeles mansion as a swinging jazz combo played just a few feet away. The occasion was a recent news conference announcing plans for this June's 28th annual Playboy Jazz Festival.
Hefner did what he could to stay in the background while others announced a lineup of stars for the June 17-18 event at the Hollywood Bowl that is to include George Duke, Stanley Clarke, Chuck Mangione, Elvis Costello, Kevin Eubanks, Allen Toussaint, the Spanish Harlem Orchestra and numerous others. Then, after a brief photo op with some of the performers, he tried to slip hurriedly back into his lavish Playboy Mansion, the turreted stone edifice that visitors who arrive by car are advised to approach slowly as a sign warns them "Playmates at Play."
But buttonholed briefly at his back door, the man who gave the world fold-out photos of beautiful naked women becomes downright effusive when the topic of jazz is raised.
"It's a combination of music from many, many sources; a combination of Afro and Caribbean and Cuban sounds ... mixed in with particularly American sounds," he offered.
Such music, Hefner added, does much more than entertain people. In its best moments, he said, it can bridge racial and cultural gaps, bringing people of all backgrounds together.
"And I'm in favor of anything that breaks down the walls," said the creator of the swinging, free-love 1960s Playboy philosophy.
Hefner turns 80 on April 9, and almost every one of those years is reflected in a pale, deeply lined face and thinning gray hair that is turning white around the fringes.
But when the band starts to play, he moves to the music. And when the Playboy Jazz Festival's veteran producer, George Wein, breaks into a song, Hefner grins, nods knowingly and sings along.
"I can't believe it; 28 years," he says of the festival that began as a "one-shot deal" in 1979 but proved so popular he decided to hold it the next year. And then the next year and the next year after that.
"When you're having fun, time flies," he adds with a wry smile.
This year's festival will contain a special tribute to New Orleans, which Wein says is particularly important in the wake of Hurricane Katrina's devastation of the city where jazz was born.
He hopes performances at June's 28th annual Playboy Jazz Festival by such artists as Branford Marsalis and the Preservation Hall Jazz Band will bring attention to efforts to aid musicians still homeless months after the hurricane.
"We have to bring them back to play," Wein said. "We have to get homes for these musicians."
Organizers note that arguably every major jazz figure of his or her time has appeared at the Playboy Jazz Festival, from Count Basie to Benny Goodman, Etta James to Lionel Hampton, Bobby McFerrin to this year's 26-year-old Japanese wunderkind, Hiromi.
When jazz greats Duke and Clarke decided to re-team for their first world tour together in 16 years, Duke says, it was only natural they would make this year's festival one of their first stops.
"We want to let people know we're back out there," Duke explained.
At the same time, the festival has branched out over the years to include far more than just traditional jazz.
Wein notes that rocker Costello, for example, will be teaming with New Orleans fusion musician Toussaint this year in a collaboration that was born of a recent benefit appearance on behalf of Katrina victims.
Also represented will be Senegal's Baaba Maal, whose music ranges from traditional African folk to international dance. Latin pianist Eddie Palmieri will lead the Afro-Caribbean Jazz All-Stars.
Such a potpourri represents a natural evolution for the festival, says Hefner, who adds that jazz itself continues to borrow from, and lend to, various musical styles.
"I think music changes, tastes change," he said, sipping a Pepsi on ice. "But from the beginning, it doesn't matter what you call it. It's always good music."