He has spoken to his successor, current Director Mark Sullivan, about the matter and says his response to the matter is the same. "It's a combination of disbelief and rage rolled into one."
In a phone interview with CBS News, Stafford says he understands the "media frenzy" about the allegations, but thinks it unfair that it tarnishes the reputation of the 99.9% of Secret Service personnel who had nothing to do with it.
He says Secret Service personnel would make the ultimate sacrifice in the line of duty but "the allegations about our culture are untrue and unfair."
Even now, Stafford says he'll be "flabbergasted" if the allegations of Secret Service partying and consorting with prostitutes while in Cartagena prove true.
Stafford served in the Secret Service details of six U.S. presidents including Nixon, Ford, Carter, Reagan, Bush (41) and Clinton. He rose to head of the Presidential Protective Division and served as Secret Service Director from 1999 to 2003 under Presidents Clinton and George W. Bush.
Stafford doesn't think that current Director Sullivan has any cause to resign. He says Sullivan is a career agent who has demonstrated "zero tolerance" for the kind of misconduct alleged to have taken place last week in Colombia.
"Unless a leader can't lead, there's no reason for resignation," said Stafford. The White House said yesterday that Pres. Obama retains confidence in Sullivan.
The allegations of misconduct are under investigation. Stafford hasn't been informed of the identities of the personnel involved. But the consequence could be severe, he said, including suspension and termination.