The 6-3 ruling effectively shaves four years off the 16-year sentence of a former police officer who was convicted of sexually abusing his son.
It is the latest in a series of high court rulings over the past seven years that limits judges' discretion in sentencing defendants. The court has held repeatedly that a judge may not increase a defendant's sentence based on factors that were not determined by a jury.
"This court has repeatedly held that, under the Sixth Amendment, any fact that exposes a defendant to a greater potential sentence must be found by the jury, not a judge, and established beyond a reasonable doubt, not merely by a preponderance of the evidence," Justice Ruth Bader Ginsburg wrote for the court, referring to an article in the U.S. Constitution.
Several states have changed their sentencing laws to require prosecutors to prove to a jury aggravating factors that could lead to longer sentences.
The court essentially followed its own precedent with this ruling, according to CBS News legal analyst Andrew Cohen.
"This does not change the law except in California and is part of a pretty firm trend in this direction," Cohen said.
California had argued that a 2005 state Supreme Court decision interpreting the state's Determinate Sentencing Law effectively brought the state into compliance with the U.S. high court's rulings. The law instructs judges to sentence inmates to the middle of three options, unless factors exist that justify the shorter or longer prison term.
The state warned that its criminal justice system would be burdened by having to re-sentence thousands of inmates.
Rather than prescribing a way to fix the law, Ginsburg said, "The ball lies in California's court."
Justice Samuel Alito said in dissent that California's law "is indistinguishable in any constitutionally significant respect" from the federal sentencing guidelines that have been approved by the Supreme Court.
There were just under a quarter-million felony convictions in the state in 2005. Data from the 1980s cited by the California Supreme Court suggests that roughly 15 percent of cases involving just one felony count result in sentences in which a judge, not a jury, finds an aggravating factor to justify the additional punishment.
But Peter Gold, a lawyer representing the former police officer, told the court that in many cases the standard term and longer option differ by just a year. In practical terms, many of those who might be affected by Wednesday's ruling might already have finished serving their time in prison.
Several justices suggested during oral argument in October that the state could tweak the law, rather than overhaul it, to remove constitutional violations.
Nine other states urged the court to uphold the California law.