Michael Magidson, Jose Merel and Jason Cazares, all 24, were charged with beating and strangling a 17-year-old who was born Edward Araujo but lived as a girl named Gwen.
One defense lawyer had argued that it was a case of manslaughter rather than murder because of his client's shame and revulsion at discovering he had had sex with a man.
Jurors said they were deadlocked 10-2 in favor of acquitting Merel and Cazares on first-degree murder charges and 7-5 in favor of convicting Michael Magidson.
Prosecutor Chris Lamiero said he plans to retry the case.
According to testimony, Merel and Magidson had had sex with Araujo and became suspicious after comparing notes. Araujo was killed in 2002 after a confrontation in which her biological identity was exposed.
The case has been closely watched by transgender advocates, who said the verdicts would send a message about how much their lives are valued.
Superior Court Judge Harry Sheppard declared the mistrial after the foreman announced that the eight men and four women were unable to reach a verdict.
Some Araujo supporters gasped in dismay, and Araujo's sister hurried out of the courtroom in tears.
"We're astonished that the jury could not convict on murder at this point. We believe that there is enough evidence that there was a murder that took place," activist Julie Dorf said.
Araujo's family had no immediate comment.
Magidson's attorney, Michael Thorman, said it appeared some jurors agreed that sexual provocation led to the killing.
Thorman acknowledged his client played a role in the attack but said the slaying was no more than manslaughter because it was provoked by Magidson's "shame and humiliation, shock and revulsion" upon learning he had had sex with a man.