Bergin, a longtime friend of Carolyn Bessette's, said his affair with Bessette as a married woman began in Los Angeles in 1997, when she spent 11 days with him after saying Kennedy was on a kayaking trip with his friends.
"It just seemed like it was old times," he said. "Like ... we were together again. It was very surreal."
Bergin was living in Hawaii in July 1999, when Kennedy and Bessette were killed in a plane crash off the coast of Martha's Vineyard.
"I never really got a chance to say goodbye," he said. "Oh, God. I just, uh, I blocked so much of it out of me, for all these years and it's just, I think I'm reliving it right now."
The interview airs Monday night on A&E.