The Rev. Al Sharpton called the decision "a shocking display of how the judicial system continues to fail to protect citizens from police abuse."
The NAACP urged federal prosecutors to appeal the ruling.
"This was clearly one of the worst cases of police brutality in the history of our nation," said NAACP President Kweisi Mfume. "I am alarmed and disappointed by the court's decision. It represents a miscarriage of justice."
Louima, at his home in Miami, had no comment.
The officers have insisted on their innocence in one of the nation's most shocking brutality scandals.
As part of its ruling, the 2nd U.S. Circuit Court of Appeals ordered that one of the defendants, Charles Schwarz, who was found guilty of holding Louima down while he was sodomized by another officer with a broken broomstick, be retried on civil rights charges.
The appeals court ruled that a defense attorney's conflict of interest denied Schwarz, 36, effective representation. The court also found that the jury was improperly exposed to prejudicial information during deliberations that was not part of the evidence introduced at trial.
The three-member appeals panel reversed the convictions of the other defendants Thomas Wiese, 38, and Thomas Bruder, 35, for obstructing a grand jury investigation by lying to protect Schwarz. It ruled there was insufficient evidence to convict the men and that they cannot be retried on the charge.
"You have nothing without your freedom, and it means everything to me," Wiese told WCBS TV. "And it's just the greatest day, a rebirth, that I got it back."
The ruling did not affect the guilty plea of the main attacker, Justin Volpe, 37, who admitted he sodomized the handcuffed Louima with a broken broomstick in a fit of rage. Volpe is serving 30 years.
"In this case, the guilty plea of Justin Volpe demonstrated that this did happen. Abner is a victim, and as long as it takes, we will cooperate with the federal government to see to it that justice is done," Louima's lawyer, Stanford Rubenstein said.
The appeals court entered a judgment of acquittal for all three officers on the obstruction charges, effectively bringing an end to the case against Wiese and Bruder. The two men had been given five-year prison sentences but have been free on bail during their appeal.
However, the court ordered a new trial on the civil rights charges for Schwartz, who is serving 15 years behind bars in Oklahoma.
Louima had been arrested in a melee outside a Brooklyn nightclub. According to testimony, Volpe was enraged because he believed Louima had punched him from behind. Louima was brutalized in the bathroom and spent two months in the hospital with a ruptured bladder and colon.
Schwarz's wife, Andra, said the family is looking forward to having him home. "It's like a dream," she said. He could be freed on bail as early as next week, his lawyer said.
"It's a sweet day when you can show the government was wrong and it was wrong from the beginning," said Stuart London, Bruder's lawyer.
Joseph Tacopina, Wiese's attorney, said his client wants to "resume his normal life and possibly return to the force."
Schwarz has denied ever being in the bathroom. Even after his conviction, he insisted that Louima and the government's other star witness, a fellow officer, confused him with Wiese. Volpe himself indicated Schwarz was not there.
In its ruling, the appeals court suggested that Schwarz's attorney at the time, police union lawyer Stephen Worth, did not call Volpe as a witness because he wanted to avoid implicating Wiese, a union delegate.
The court said there was a "distinct possibility" that "Worth would sacrifice Schwarz's interests" for those of the police union.
Worth did not immediately return a call for comment.
Louima sued the city and the police union and settled in July for $8.7 million - the largest payout in a police brutality case in New York.
The 1997 racially charged attack on Louima while he was in custody at the 70th Police Precinct in Brooklyn sparked demonstrations, mostly by black and Hispanic residents who said they were unfairly targeted by officers of the nation's largest police force.