But now, there is another woman in his life: prosecutor Janet Albertson.
"Danny Pelosi is a drinker, a gambler," says Albertson. "He's a petty thief."
As the trial begins, all she has is a lot of circumstantial evidence to offer as proof that Pelosi killed for money.
So who says Pelosi was at the murder scene? According to Albertson, Pelosi himself. She says he confessed to a number of people: "Stupid people do stupid things. That's why we catch them."
Albertson calls a contractor to the stand who worked with Pelos who says a year before the murder, Pelosi talked to him about killing Ammon. One of Pelosi's ex-girlfriends also testifies that Pelosi admitted to the murder.
But Pelosi's own father may have been the most damaging witness of all. He testifies that Pelosi asked him, just hours after the murder, how to dispose of something where no one could ever find it.
The prosecution thinks Pelosi was trying to get rid of evidence from the crime scene, especially the hard drive with pictures from the night of the murder.
"That unit recorded 24 hours a day, so of course it would have recorded the image of the murderer, coming in, going out, removing all the evidence," says Albertson.
But for jurors, it was what Pelosi did and said to his father as he left the courtroom that was the most telling. He snarled at his father in front of the jury and said, "I hate him."
"That had a tremendous effect on me," says one juror. "Not just his relationship with his father, but the anger and rage that just suddenly came up like that."
Pelosi's attorney argues that his client had a love-hate relationship with his father. He also tells the jurors that each of the witnesses had a reason to lie about Pelosi's confession.
But another relative, Pelosi's sister, Barbara, was subpoenaed by the prosecution. Generosa had hired her to monitor the surveillance system on her laptop and spy on Ammon in the mansion.
Barbara says she saw something suspicious earlier in the evening, during a routine check, and that when Pelosi arrived the night of the murder, she asked him to log on and check it out. "I think I hurt Danny," says Barbara. "Because I told him to go on the computer that night. Because he honestly did not ask me to turn it on."
But Albertson says Pelosi had his own reasons for logging on that night: "I think what he wanted to know was when Ted Ammon was at the house. I think he also had an interest in whether anyone else was there."
Shargel, however, set out to prove that there is no evidence Pelosi was even at the crime scene: "Ted Ammon died at a time when Danny Pelosi was not on Eastern Long Island. And as far as I'm concered, that's the case."
The night of the murder, Ammon went out for dinner, paid for his meal and left around 9:30 p.m. At 9:44 p.m., he made what could be his last phone call, to his girlfriend. He said he saw something that scared him and he was heading back home.
That phone call is the cornerstone of Shargel's defense. He believes that someone could have followed Ammon home from the beach and killed him: "To have what seemed to be a confrontation, or a near confrontation, I think, is significant."
The timing of the call fits neatly into Shargel's theory. According to his star witness, Dr. Werner Spitz, a world-renowned scientist, there was still enough undigested food in Ammon's stomach to prove he could not have lived more than four hours after eating.
Spitz insists Ammon was dead by 1 a.m., and Shargel proves that Pelosi was miles away from East Hampton during that time. His cell phone records show he was driving from Manhattan to his sister's house.
Medical experts brought in by the prosecution, however, say Ammon could have been alive much longer, giving Pelosi time to get there.
As the trial wears on, all eyes turn on Pelosi himself. He told 48 Hours in a phone call from jail that he's feeling the pressure: "I'm afraid to look at the jury. I'm afraid to do anything, because I'm too emotional, everyone says. I'm too emotional."
Pelosi won't take the stand in his own defense. But can he really keep quiet? "I'm going to follow the advice of my attorneys because it looks like I don't even have to testify at this point, because the truth is the truth," says Pelosi.
On the day that Shargel hoped to hand his case over to the jurors, there's a surprise witness, one Shargel never wanted to call.
Pelosi, against Shargel's advice, had ecided to take the stand. "My mouth dropped to the floor," says Shargel. "I couldn't believe he took the stand."
Pelosi told 48 Hours by phone from jail that he knew what was at stake from the moment he took the oath. "When I got on the stand, the first thing I had to do was get my heart back in my jacket," he says.
Albertson says she was hoping to get a rise out of Pelosi.
"She got under my skin a few times," says Pelosi. "And I know I came back a little arrogant a few times, but the woman was attacking me."
While Pelosi was on the stand, he dropped what he called a bombshell and implicated his late wife, Generosa. He says she asked him to kill Ammon for her, but he refused. He also said Generosa's alibi, that she was in her Manhattan apartment the night of the murder, was a lie. In fact, he says she left the apartment before he did.
"This woman told me a hundred times that she was going to go kill her husband," says Pelosi. "Maybe this time it happened."
On the stand, Pelosi only suggested Generosa was involved. But later, in a phone conversation, he went further. "I kept my mouth shut because my wife, right," he says. "Generosa married me to keep my mouth shut. That was the whole reason, to keep my mouth shut."
Pelosi also says the evidence shows Generosa did it, but not alone. And that he was not involved: "I had nothing to do with it."
In his closing argument, Shargel asked repeatedly, "Where was Generosa Ammon that night?"
"I don't think she was there alone," says Shargel. "But she wasn't there with Danny Pelosi."
Finally, after more than two months of testimony, the jurors got the case late on a Friday night. All throughout the weekend, both sides camped out and waited.
Inside the jury room, the jurors were troubled by the lack of physical evidence. And that was reason for Pelosi to be optimistic. "If they do their job properly, and they weigh the evidence, and they listen to everything that was said, I'm going to be sitting home for Christmas with my son that I still haven't seen," he says.
After three long days of deliberating, the jury reached a verdict. Pelosi was convicted of second-degree murder. When he heard the verdict, he started softly crying.
"When I heard guilty, I was in shock," says Pelosi.
In the end, the jurors couldn't ignore the missing hard drive that could have recorded images from the murder. And, there was the matter of the millions in fortune.
Jurors believe Pelosi was worried that he might lose access to the money if Generosa signed a proposed divorce settlement, as scheduled.
But jurors still believe there's room for more suspects. In fact, some jurors still aren't sure if Pelosi was there at all.
But in the end, the jury voted that whatever his exact role was that night, Pelosi intended to kill Ammon, and took part in the murder.
In January, Pelosi was sentenced to 25 years to life in prison.
"I'm not doing it for 25 years. Listen to me loud and clear," says Pelosi. "I'm appealing this issue, and on the appellate circuit, it will be heard. I guarantee it."
The Ammon twins are now 15 years old. They live in the South with Ted Ammon's sister.