The ruling overturned the death sentence of Johnny Paul Penry, whose lawyers claim their client has the mind of a 7-year-old, still believes in Santa Claus and likes to play with coloring books.
The case, sent back to a federal appeals court, does not answer a larger question about whether execution of the mentally retarded is constitutional.
Penry killed Carpenter with her own pair of scissors after forcing his way into her home and raping her. He was out on probation for a 1977 rape at the time.
Penry's IQ is said by his lawyers to be between 50 and 63, below the 70 required for normal intelligence. Prosecutors have argued that Penry, who confessed to the crime, has been pretending to be retarded.
Penry's attorneys argued the judge in the case issued "hopelessly confusing instructions" to jurors before they began deciding whether Penry should live or die.
The opinion by Justice Sandra Day O'Connor said the jury instruction was "ineffective and illogical" and did not allow jurors to consider mitigating evidence of Penry's mental retardation and childhood abuse.
Penry then was retried, convicted and again sentenced to death in 1990.
But in the resentencing, O'Connor wrote, the Texas trial court did not follow the Supreme Court's guidance.
O'Connor said the new instructions failed to provide the jury "with a vehicle for expressing its reasoned moral response to the mitigating evidence of Penry's mental retardation and childhood abuse."
The second sentencing as did the first left no vehicle for jurors to express the view that Penry should get life, not death, based on mitigating evidence, O'Connor said.
O'Connor wrote that jurors were asked to vote "no" to specific questions if they thought the death penalty was inappropriate even if their answers to those questions would have been "yes."
"The jury was essentially instructed to return a false answer in order to avoid a death sentence," O'Connor wrote.
The vote was 6-3 on the crucial question of the instructions. Chief Justice William Rehnquist and Justices Antonin Scalia and Clarence Thomas the court's three most conservative members dissented from the ruling.
The high court will consider the broader issue of executing mentally retarded people iits term that begins in October, in another case.
In its 1989 ruling in the Penry case, the Supreme Court found that executing the mentally retarded did not violate the cruel and unusual punishment ban.
The Texas Legislature recently passed a bill banning the execution of mentally retarded persons. Gov. Rick Perry hasn't said whether he will sign it. Texas leads the nation in executions.
State Sen. Rodney Ellis, who sponsored the bill, has said six mentally retarded people have been put to death in Texas since the state resumed executions. State Rep. Juan Hinojosa says seven retarded inmates are on death row.
In another section of the decision, the court unanimously held that admission into evidence of statements from a psychiatric report based on an interview with Penry did not violate his constitutional right against self-incrimination.
© MMI Viacom Internet Services Inc. All Rights Reserved. This material may not be published, broadcast, rewritten, or redistributed. The Associated Press and Reuters Limited and contributed to this report