The 90-year-old Stevens said it is wrong to lump all Muslims with the terrorists who carried out the Sept. 11 attacks that killed 3,000 people. "Guilt by association is unfair," he told a Japanese-American group in Washington.
The center's location two blocks north of where the Twin Towers once stood has upset some relatives of Sept. 11 victims and stirred nationwide debate and angry demands that it be moved. Critics say the site of mass murder by Islamic extremists is no place for an Islamic institution, while supporters of the center say religious freedom should be protected.
More Islamic Center Coverage
Saudi Prince Backs Move for NYC Islamic Center
NY Islamic Center Imam, Wife Have Police Contact
NYC Islamic Center Imam, Wife Get Death Threats
"60 Minutes" - The Heated Debate Over NY's Islamic Center
Muslims: "Ground Zero Belongs to All Americans"
But Stevens, a World War II veteran, compared the criticism of the mosque to the emotion he said he initially felt when he saw Japanese tourists at Pearl Harbor.
Among the thoughts that he said flashed through his mind during a 1994 visit to the memorial to the Japanese attack that brought the U.S. into World War II was, "These people don't really belong here."
He said many New Yorkers might have had a similar reaction to news about the mosque in lower Manhattan.
But Stevens said he realized he was drawing conclusions about a group of people that did not necessarily fit any one of the tourists he saw at Pearl Harbor.
"We should never pass judgment on barrels and barrels of apples just because one of them may be rotten," said Stevens, who left the court in June. He commented on an issue of public debate in a way he most likely would have avoided had he still been serving as a justice.
He said that a nation built by people who fled religious persecution "should understand why American Muslims should enjoy the freedom to build their places of worship wherever permitted by local zoning laws."
Stevens said the National Japanese American Memorial in Washington offers a similar message in its recognition that the internment of thousands of U.S. citizens of Japanese descent during World War II was wrong.
He called the monument a "a powerful reminder of the fact that ignorance - that is to say, fear of the unknown - is the source of most invidious prejudice."