Proceeds from a sale of the property and its contents could be used to help reimburse those who lost billions of dollars investing with Madoff before he confessed to running a Ponzi scheme.
U.S. Marshal Joseph Guccione said the marshals arrived at the property at noon with a court order permitting them to take custody of the apartment and to make anyone living there move out.
Guccione said Madoff's wife Ruth had been advised in advance of the marshals' plans and was leaving the residence and surrendering all personal property.
"She will be leaving," he said at midday. "Restitution for the victims is the government's top priority."
A law enforcement source told CBS News investigative producer Pat Milton that Ruth Madoff was reluctant to leave the apartment and pleaded with the marshals to allow her to keep her fur coat. They told her no.
Marshals will be changing the locks, but they do not plan to remove any items today, CBS News has learned.
By about 1 p.m., the 67-year-old Ruth Madoff had left. It was not immediately clear where she went to live.
"Ruth moved out voluntarily pursuant to the prior agreements we reached with the government," said her laywer, Peter Chavkin.
According to the New York Post, Madoff's next move is unclear because several Manhattan buildings have refused to rent to her.
The 71-year-old Madoff was sentenced Monday to 150 years in prison. He pleaded guilty in March to charges that his investment advisory business was a multibillion-dollar Ponzi scheme that wiped out thousands of investors and ruined charities.
Authorities said Madoff had carried out the fraud for at least two decades before confessing to his sons in December that his investment business was a fraud and that he had lost as much as $50 billion.
Last week, Ruth Madoff agreed to give up all of her possessions in return for a promise that federal prosecutors would not pursue $2.5 million not tied to the fraud. The money, though, is not protected from civil legal actions that might be pursued by a court-appointed trustee liquidating Madoff's assets or by investor lawsuits.
when she said in a statement that her husband "stunned us all with his confession and is responsible for this terrible situation in which so many now find themselves."
Before she agreed on a deal with the government to resolve her finances a week ago, Ruth Madoff had indicated through lawyers that she planned to try to keep the penthouse and an additional $62 million in assets as unrelated to the fraud.
Before the fraud was exposed, the Madoffs had homes in Palm Beach, Fla., the south of France and the tip of Long Island along with the midtown Manhattan penthouse. They also traveled by private jet and yacht.
The couple met at their Queens high school and married in 1959. Ruth Madoff worked with her husband when he started his financial business in 1960 and she reportedly still had an office near his when the fraud was exposed.
Madoff has said he operated his fraud without the knowledge of his family.
Complete Bernard Madoff coverage:
Madoff Penthouse Property Seized
Feds Cast Wider Net In Madoff Case
Madoff Feels Remorse, Lawyer Says
10 More To Be Charged, Source Says
Madoff Sentenced To 150 Years In Prison
Ruth Madoff: "Embarrassed And Ashamed"
Madoff's Fraud: A Family Affair?
Transcript of Madoff Sentencing
Analysis: 150-Year Sentence "Grossly Unfair"
Court Sketches: Madoff Sentencing