Rene-Thierry Magon de la Villehuchet and his business partner Patrick Littaye were "totally ruined," Bertrand Magon de la Villehuchet said in a telephone interview from his home on Paris' chic Place des Vosges.
Bertrand, 74, said his brother had "invested his own fortune" with Madoff - up to several tens of millions of dollars - along with money from friends and family.
Rene-Thierry, 65, was found dead at his desk in the New York office of Access International Advisors on Tuesday, both of his wrists slashed. A box cutter and a bottle of sleeping pills lay nearby. Police say it was a suicide.
Rene-Thierry had begun investing with Madoff three or four years ago and had a total of $1.4 billion invested with him when the scandal came crashing down, according to his brother.
"At first he thought he'd be able to get the money back. He was very determined. Gradually he realized he wouldn't be able to," Bertrand said.
"He trusted Madoff completely," he said.
Madoff, 70, was arrested Dec. 11 and allegedly told FBI agents he had masterminded a $50 billion Ponzi scheme that ensnared investors far and wide, from retirees to charities to the International Olympic Committee.
Madoff, a former chairman of the Nasdaq stock market, has since become one of the most vilified people in America.
Rene-Thierry's fund was among the biggest losers in the scheme, and one of a handful to get taken for more than $1 billion.
Famous names reported to have lost their investment with him include L'Oreal cosmetic empire heiress Lilliane Bettencourt, listed by Forbes as the world's richest woman. Bertrand declined to identify any of his brother's investors outside the family.
Bertrand said he spoke with his brother almost every day. "My brother was a man of simple tastes," he said. "He was a very modest man.
"A lot is being said about him, like that he flew in by helicopter to his chateau - that's not true," Bertrand added.
Bertrand said he'll be joining class action lawsuits against Madoff and the Securities and Exchange Commission.
Rene-Thierry raised the money he placed with Madoff from friends and family. Bertrand said he lost 20 percent of his own savings in the scam.
Contrary to what some analysts say now, Bertrand said the returns on his investment with Madoff weren't too good to be true.
"Over four years my gain was 17 percent - that's not crazy," he said.
The Magon de la Villehuchet family built its fortune in shipping during the 17th century according to Georges le Gorgeu, a historian in Plouer-sur-Rance, where the clan has owned a chateau for several hundred years.
The family was so rich and prominent that it loaned money to France's Sun King, Louis XIV, who ennobled them, le Gorgeu said. Many of Rene-Thierry's and Bertrand's aristocratic ancestors died on the guillotine during the French revolution, le Gorgeu said.
Rene-Thierry, who had married and had one sister, spent every August at the chateau, his brother said. In recent years he'd invested a considerable sum to renovate the chateau and protect its archives, which date back to 1200, le Gorgeu said.