The former longtime Republican, who switched party allegiances in April, was one of several Democrats to face heated criticism from constituents in town hall meetings across the country over proposed health care reform. (See video below.)
Opponents have railed against the Democratic proposals, characterizing them in general as government-run health care that will overburden taxpayers. Critics have taken specific issue with so-called "death panels," which will supposedly force elderly Medicare patients to seek end of life treatments. (Though those rumors have been proven false.)
"These people have a right to be organized," Specter said on CBS' "The Early Show" Wednesday. "I'm not going to complain about the fact that they are organized. They have a right to do that and they have a right to speak. But I think we ought to understand that they're not necessarily representative of America."
But Specter said that he's been "impressed with the fact that people [are] very well prepared," even though he has "no doubt the materials have been supplied to them."
"They've come in with copies of the House bill, with the sections marked. And also, although they may not be representative, they're significant. And they have to be listened to. And I think that it's a loud, clear warning to Congress, to Washington, that we've got to do something about it. And not just continue the same old partisan wrangling in Washington.
Specter said the frustration over health care may be linked to deeper problems in the country.
"I think there is a mood in America of anger. With so many people unemployed and so much bickering in Washington, people are disgusted with the partisanship and with the fear of losing their health care, it all boils over."
And Specter did give credence to opponents' concerns over adding to an already hefty deficit – an issue he says will be addressed in whatever legislation Congress produces.
"I think President Obama was wise to make it explicit he wouldn't sign a health care bill that added to the deficit. And I have said publicly at these meetings repeatedly I will not vote for a health care plan that produces a deficit. So there is a great concern about that."