A three-judge panel of the 9th U.S. Circuit Court of Appeals issued the unanimous decision in the dispute over the 29-foot (9-meter) cross, which was dedicated in 1954 in honor of Korean War veterans.
The court said modifications could be made to make it constitutional, but it didn't specify what those changes would be.
"In no way is this decision meant to undermine the importance of honoring our veterans," the three judges said in their ruling. "Indeed, there are countless ways that we can and should honor them, but without the imprimatur of state-endorsed religion."
Federal courts are reviewing several cases of crosses on public lands being challenged as unconstitutional, including a to honor American war dead. Tuesday's ruling could influence future cases involving the separation of church and state.
U.S. Justice Department spokesman Wyn Hornbuckle said the federal government, which is defending the San Diego cross, is studying the ruling and had no comment.
Gina Coburn, spokeswoman for the San Diego's city attorney's office, which was once a defendant in the case, said the cross will have to be removed from Mount Soledad unless a full panel of 9th Circuit judges reverses Tuesday's decision or the Supreme Court agrees to rule on it.
The American Center for Law and Justice, an evangelical Christian legal group which has fought to preserve the cross, called it "a judicial slap in the face" to military veterans and said they planned to ask the Supreme Court to intervene.
The legal fight over the Mount Soledad cross began in 1989 when atheist Philip Paulson sued the city of San Diego. Paulson, a Vietnam War veteran, contended that the cross excludes veterans who aren't Christian. A Jewish war veterans group has also been a plaintiff in the case along with the American Civil Liberties Union.
State and federal judges have ordered the cross removed, saying it represents an unconstitutional endorsement of one religion. But in 2006, the U.S. Supreme Court blocked an order that the city take it down that summer, giving lower courts time to hear appeals.
City officials have argued that the cross is part of a secular war memorial, and the cross has been embraced by San Diego residents who in 2005 overwhelmingly approved a measure to preserve it by donating it to the federal government.
The land under the cross was eventually transferred to the federal government but the courts have said that did not protect it from the constitutional dispute.
Joe Infranco, senior counsel of The Alliance Defense Fund, an Arizona-based Christian legal group, said the memory of troops should not be dishonored because the ACLU and a few others are offended by the presence of the cross.
"It's tragic that the court chose a twisted and tired interpretation of the First Amendment over the common sense idea that the families of fallen American troops should be allowed to honor these heroes as they choose," he said.
David Blair-Loy of the ACLU in San Diego County said the court acted correctly.
"We honor those who have served, but the Constitution does not allow the government to exclude non-Christians by endorsing a clearly religious symbol," he said.
The Rev. John Fredericksen of Orlando, Fla., was among a steady stream of people who visited the white cross Tuesday atop Mount Soledad, which affords spectacular views of the Pacific Ocean and the surrounding upscale suburb of La Jolla.
"For those who are offended, they can move or look somewhere else," the 56-year-old Christian pastor said. "Christians are not asking every mosque or synagogue to be torn down. Why tear down a symbol of Christianity? Let them find or make their own memorial."
Michael Aguirre, a former San Diego city attorney who has followed the case closely, said cross supporters will have to counter the court's analysis that the cross was used historically to promote Christianity.
The ruling recounts that the cross was dedicated on Easter Sunday and used for religious gatherings for nearly three decades before it became a war memorial. It said La Jolla has a "well-documented history" of anti-Semitism from the 1920s to around 1970.
"This cross marks La Jolla as a Christian community, that's basically what (the judges are) saying," said Aguirre, who is now in private practice. "It was a cross for decades in a community with a history of anti-Semitism."