Well, actually, it was with women. But the founding editor of Playboy magazine also fell in love early with jazz, and both affairs have lasted a lifetime.
"It's the music of my youth. It's the music I grew up with. It's the music of my dreams," Hefner, looking wistful, said as he stood in the backyard of his Los Angeles mansion as a swinging jazz combo played just a few feet away. The occasion was a recent news conference announcing plans for this June's 28th annual Playboy Jazz Festival.
Hefner did what he could to stay in the background while others announced a lineup of stars for the June 17-18 event at the Hollywood Bowl that is to include George Duke, Stanley Clarke, Chuck Mangione, Elvis Costello, Kevin Eubanks, Allen Toussaint, the Spanish Harlem Orchestra and numerous others. Then, after a brief photo op with some of the performers, he tried to slip hurriedly back into his lavish Playboy Mansion, the turreted stone edifice that visitors who arrive by car are advised to approach slowly as a sign warns them "Playmates at Play."
But buttonholed briefly at his back door, the man who gave the world fold-out photos of beautiful naked women becomes downright effusive when the topic of jazz is raised.
"It's a combination of music from many, many sources; a combination of Afro and Caribbean and Cuban sounds ... mixed in with particularly American sounds," he offered.
Such music, Hefner added, does much more than entertain people. In its best moments, he said, it can bridge racial and cultural gaps, bringing people of all backgrounds together.
"And I'm in favor of anything that breaks down the walls," said the creator of the swinging, free-love 1960s Playboy philosophy.
Hefner turns 80 on April 9, and almost every one of those years is reflected in a pale, deeply lined face and thinning gray hair that is turning white around the fringes.
But when the band starts to play, he moves to the music. And when the Playboy Jazz Festival's veteran producer, George Wein, breaks into a song, Hefner grins, nods knowingly and sings along.
"I can't believe it; 28 years," he says of the festival that began as a "one-shot deal" in 1979 but proved so popular he decided to hold it the next year. And then the next year and the next year after that.
"When you're having fun, time flies," he adds with a wry smile.