"It never entered the SEC's mind that it was a Ponzi scheme," Madoff said, because of "the reputation I had."
"They thought the likelihood of Madoff being a big criminal was probably not something that was realistic," said Paul Atkins, a former SEC Commissioner.
At the height of his career, Madoff was regarded as a financial genius, he even served as chairman of NASDAQ, reports CBS News correspondent Randall Pinkston.
In 2003, Madoff was sure he would be caught and was surprised when investigators did not check his accounts to see if he had actually traded stocks - which he had not.
It is accounting 101, Madoff told the inspector general, to look at DTC - Depositor Trust Commission - to discover a Ponzi scheme.
"With one phone call they could have brought the whole thing down," Atkins said.
"I worried every time," Madoff said in the interview. "I wish they caught me six years ago, eight years ago."
Atkins said part of the lesson is for investors to be more skeptical.
"They need to ask questions and in the Madoff case, there were not just red flags for the SEC, there were red flags for investors," Atkins said.
Boston accountant Harry Markopolos did ask questions, but his warnings to the SEC - that Madoff was running a Ponzi scheme - were ignored.
Madoff described Markopolos as "a joke in the industry."
By last December, the joke was on Madoff when he admitted that his operation was fraudulent and had cost his clients billions of dollars.
Attorney Barry Lax says that Madoff's statement is one more reason the SEC should be held accountable.
"We're representing many victims against the SEC," Lax said. "We believe the sec was simply negligent in its investigations and examinations of Madoff's business."
In a related matter, on Tuesday, David Friehling, Madoff's accountant, is expected to enter a guilty plea on fraud charges. Prosecutors say Friehling never performed an audit of Madoff's company.