Patterson died at his home, having suffered from Alzheimer's disease for about eight years, according to nephew Sherman Patterson. Patterson also had prostate cancer.
Patterson's career was marked by historic highs and humiliating lows.
He emerged from a troubled childhood in Brooklyn to win the Olympic middleweight championship in 1952.
In 1956, the undersized heavyweight became at age 21 the youngest man to win the title with a fifth-round knockout of Archie Moore.
But three years later, Patterson was knocked down seven times in the third round in losing the title to Ingemar Johansson at the Polo Grounds in New York.
Patterson returned with a vengeance at the same site in 1960, knocking out Johansson with a tremendous left hook to retake the title.
"They said I was the fighter who got knocked down the most, but I also got up the most," Patterson said later.
Despite his accomplishment, he was so humiliated when he lost the title on a first-round knockout to Sonny Liston in 1962 that he left Comiskey Park in Chicago wearing dark glasses and a fake beard. Patterson again was knocked out in the first round by Liston in 1963.
Patterson got two more shots at winning the title a third time. Battered and taunted for most of the fight by Muhammad Ali, Patterson was stopped in the 12th round in 1965. He lost a disputed 15-round decision to WBA champion Jimmy Ellis in 1968.
Overall, Patterson finished 55-8-1 with 40 knockouts. He was knocked out five times and knocked down a total of at least 15 times. He was inducted into the International Boxing Hall of Fame in 1991.
After retiring in 1972, Patterson remained close to the sport. He served twice as chairman of the New York State Athletic Commission. His second term began when he was picked in 1995 by Gov. George Pataki to help rebuild boxing in New York.
On April 1, 1998, Patterson resigned the post after a published report that a three-hour videotape of a deposition he gave in a lawsuit revealed he couldn't recall important events in his boxing career.
Patterson said he was very tired during the deposition and, "It's hard for me to think when I'm tired."
Patterson, one of 11 children, was in enough trouble as a youngster to be sent to the Wiltwyck School for Boys. After being released, he took up boxing, won a New York Golden Gloves championship and then the Olympic gold medal in the 165-pound class at Helsinki, Finland.
"If it wasn't for boxing, I would probably be behind bars or dead," he said in a 1998 interview.
He turned pro in 1952 under the management of Cus D'Amato, who in the 1980s would develop another heavyweight champion, Mike Tyson. Patterson fought as a light heavyweight until becoming a heavyweight in 1956.
After regaining the title, Patterson was the verge of losing it again when he was knocked down twice by Johansson in the first round in 1961. But Patterson knocked down Johansson before the round was over, then won on a sixth-round knockout.
He made one successful defense, then lost the title to Liston in a fight a lot of people didn't want him to take. In fact, taking the match caused a split between Patterson and D'Amato.
Patterson said in 1997 that another person who didn't want him to fight Liston was President Kennedy.
"I'm sorry, Mr. President," Patterson said he told Kennedy. "The title is not worth anything if the best fighters can't have a shot at it. And Liston deserves a shot."
Patterson retired after he was stopped by Ali in the seventh round of a non-title match in 1972 at Madison Square Garden.
Patterson and his second wife, Janet, lived on a farm near New Paltz, N.Y. After leaving the athletic commission, Patterson counseled troubled children for the New York State Office of Children and Family Services.
He also adopted Tracy Harris two years after the 11-year-old boy began hanging around the gym at Patterson's home. In 1992, Tracy Harris Patterson, with his father's help, won the WBC super bantamweight championship.
Funeral services for Patterson will be private.