The two leaders of the construction project, meanwhile, defended their plans, though one suggested that organizers might eventually be willing to discuss an alternative site. The other, Imam Feisal Abdul Rauf, said during a Middle East trip that the attention generated by the project is actually positive and that he hopes it will bring greater understanding.
Around the corner from the cordoned-off old building that is to become a 13-story Islamic community center and mosque, police separated the two groups of demonstrators. There were no reports of physical clashes but there were some nose-to-nose confrontations, including a man and a woman screaming at each other across a barricade under a steady rain.
Opponents of the $100 million project two blocks from the World Trade Center site appeared to outnumber supporters. Bruce Springsteen's "Born in the USA" blared over loudspeakers as mosque opponents chanted, "No mosque, no way!"
Signs hoisted by dozens of protesters standing behind police barricades read "SHARIA" - using dripping, blood-red letters to describe Islam's Shariah law, which governs the behavior of Muslims.
Steve Ayling, a 40-year-old Brooklyn plumber who carried his sign to a dry spot by an office building, said the people behind the mosque project are "the same people who took down the twin towers."
Opponents demand that the mosque be moved farther from the site where more than 2,700 people were killed on Sept. 11, 2001. "They should put it in the Middle East," Ayling said.
"If they build a mosque in the shadow of the world trade center before we're done rebuilding, what's next?" construction worker Andy Sullivan said addressing the rally.
On a nearby sidewalk, police chased away a group that unfurled a banner with images of beating, stoning and other torture they said was committed by those who followed Islamic law.
A block away was a smaller group of supporters, reports CBS News correspondent Michelle Miller.
"How far away is okay? A mile away?" Matt Sky, who was marching in support of religious freedom, asked Miller. "Two miles?"
Caught in the middle are Muslim-Americans, including some who are 9/11 families.
"Don't they realize that this is bigotry?" asked Talat Hamdani. Her son, Mohammad, died that day rescuing others. "To scapegoat all Muslim-Americans for the acts of foreign terrorists is equivalent to scapegoating all Christians for the acts of Timothy McVeigh."
Others feel differently. Thirty years ago, Neda Bolourchi's family emigrated from Iran to California.
"To us, it is the gravesite of our loved ones," said Bolourchi, whose mother, Touri, was a passenger on the second plane to hit the twin towers.
"Please, not here," she told Miller. "Nowhere as part of your religious freedom does it guarantee you a location."
Ironically, the location has been used as a mosque for months, while the new building would not even be visible from Ground Zero, Miller reports.
A mannequin dressed in a keffiyeh, a traditional Arab headdress, was mounted on one of two mock missiles that were part of an anti-mosque installation. One missile was inscribed with the words: "Again? Freedom Targeted by Religion"; the other with "Obama: With a middle name Hussein. We understand. Bloomberg: What is your excuse?"
New York Mayor Michael Bloomberg has fiercely defended plans for the proposed mosque, saying that the right "to practice your religion was one of the real reasons America was founded."
The mosque project is being led by Rauf and his wife, Daisy Khan, who insist the center will promote moderate Islam. The dispute has sparked a story=6793375>national debate on religious freedom and American values and is becoming an issue on the campaign trail ahead of the midterm elections. Republicans have been critical of President Barack Obama's stance: He has said the Muslims have the right to build the center at the site but has not commented on whether he thinks they should.
Rauf is in the middle of a Mideast trip funded by the U.S. State Department that is intended to promote religious tolerance. He told a gathering Sunday at the U.S. ambassador's residence in the Persian Gulf state of Bahrain that he took heart from the dispute over the mosque, saying "the fact we are getting this kind of attention is a sign of success."
"It is my hope that people will understand more," Rauf said without elaborating.
Democratic New York Gov. David Paterson has suggested that state land farther from ground zero be used for the center. Khan, executive director of the American Society for Muslim Advancement, expressed some openness to that idea on ABC's "This Week with Christiane Amanpour," but said she would have to meet with the center's other "stakeholders" first.
"We want to build bridges," Khan said. "We don't want to create conflict, this is not where we were coming from. So, this is an opportunity for us to really turn this around and make this into something very, very positive. So we will meet, and we will do what is right for everyone."
But Khan also said the angry reaction to the project "is like a metastasized anti-Semitism."
"It's not even Islamophobia. It's beyond Islamophobia," she said. "It's hate of Muslims."
At the pro-mosque rally, staged a block away from opponents' demonstration, several hundred people chanted, "Muslims are welcome here! We say no to racist fear!"
Dr. Ali Akram, a 39-year-old Brooklyn physician, came with his three sons and an 11-year-old nephew waving an American flag. He noted that scores of Muslims were among those who died in the towers, and he called those who oppose the mosque "un-American."
"They teach their children about the freedom of religion in America - but they don't practice what they preach," Akram said.
John Green, who lost a friend in the attacks, said that although organizers have the right to build the project, "I think if they moved it, they would get the respect of more Americans than if they play hardball." He was demonstrating in the group of mosque opponents.
Gila Barzvi, whose son, Guy, was killed in the towers, stood with mosque opponents, clutching a large photo of her son with both hands.
"This is sacred ground and it's where my son was buried," the native Israeli from Queens said. She said the mosque would be "like a knife in our hearts."
She was joined by a close friend, Kobi Mor, who flew from San Francisco to participate in the rally.
If the mosque gets built, "we will bombard it," Mor said. He would not elaborate but added that he believes the project "will never happen."
Rauf, in an interview with Bahrain's Al Wasat newspaper,.
"American Muslims have the right to practice their religion in accordance with the Constitution of the United States," Rauf said. "I see the article of independence as more compliant with the principles of Islam than what is available in many of the current Muslim countries."
A portion of the Al Wasat interview - to be published Monday - was seen Sunday by The Associated Press.