In Washington, about 1,500 people took part in the Pentagon's walk-in tour of the site where a hijacked plane crashed into the building.
Family members worry the International Freedom Center will take attention away from those who died in the attack. They said the museum should not be allowed to show exhibits about struggles for freedom around the world.
"These are important stories to tell," said Jack Lynch, whose firefighter son Michael Lynch is one of the 2,749 people who died at the trade center. "Elsewhere, not at America's memorial."
The rally by more than a dozen family groups came a day before the fourth anniversary of the terrorist attacks. A bouquet of white flowers and several single, red roses were tucked into the metal fence surrounding the site, while huge American flags hung outside nearby office buildings.
"The IFC threatens to turn ground zero into a place of endless controversy rather than a place of honor," said Anthony Gardner, whose brother Harvey died at the trade center.
The Freedom Center and another museum, the Drawing Center, were chosen more than a year ago to occupy cultural space at the site in a building close to the planned "Reflecting Absence" memorial.
But rebuilding officials said last month the Drawing Center would look elsewhere for a home and that the Freedom Center would have to submit more detailed plans and respond to family objections to ensure its place at the site. A mediator has recently been recruited to help museum officials and the families communicate.
"We respect those family members who believe that the World Trade Center site should be limited to the memorial and memorial center museum, even as we and other family members believe that a living memorial, reflecting our common resolve to preserve freedom, should also have a place on this sacred ground," Freedom Center chairman Tom Bernstein and vice chair Paula Berry, who lost her husband on Sept. 11, said in a statement Saturday.