The trustee, Irving Picard, sought $198.7 million from Madoff's brother, Peter, who had worked at Madoff's Manhattan investment company since 1965, and sons, Mark and Andrew.
Also sued was Shana D. Madoff, Bernard Madoff's niece and Peter Madoff's daughter.
Lawyers for Picard said in papers filed in U.S. Bankruptcy Court in Manhattan that Madoff's family-run business "was operated as if it were the family piggy bank."
They said each of the family members withdrew huge sums of money to fund personal business ventures and to pay for expenses ranging from multimillion dollar homes, cars and boats to monthly credit card charges for restaurants, vacations and clothing.
The lawyers said $141 million identified as fraudulent proceeds were received by the family members in the six years before Madoff surrendered and revealed his plot last December while at least $58 million was received in the last two years.
Peter Madoff was the company's senior managing director and chief compliance officer while Mark and Andrew shared the title of co-director of trading.
Mark had worked with his father at the company since 1986 while Andrew had been there since 1988.
Shana Madoff, a lawyer, had worked there since 1995 as compliance counsel and in-house counsel, the court papers said.
The lawyers for Picard said the family members frequently held themselves out to be business and securities regulatory compliance managers and principals of Madoff's company.
"Yet the family members were completely derelict in these duties and responsibilities," they wrote.
"As a result, they either failed to detect or failed to stop the fraud, thereby enabling and facilitating the Ponzi scheme," the papers said. "Simply put, if the family members had been doing their jobs _ honestly and faithfully _ the Madoff Ponzi scheme might never have succeeded, or continued for so long."
Madoff is serving a 150-year prison sentence after admitting losing billions of dollars for thousands of clients during his half century career.