Lawyers for Phillip Garrido had argued that he is not competent to stand trial along with his wife on 18 counts each of kidnapping, rape and false imprisonment.
A jury had been set to hear arguments on the competency issue, but both sides said in court they agreed instead to let the judge decide the matter. El Dorado County Superior Court Judge Douglas Phimister quickly issued his ruling, based on numerous psychiatric reports.
Garrido, a convicted rapist, and his wife, Nancy, are accused of abducting Dugard from a South Lake Tahoe bus stop in 1991 when she was 11.
Nancy Garrido has pleaded not guilty to the charges.
District Attorney Vern Pierson said the agreement between the prosecution and defense to cancel Garrido's competency hearing before a jury would allow the case to be resolved more quickly, perhaps by the end of summer.
That would be two years after Dugard resurfaced along with her two daughters fathered by Garrido.
"She is in pretty good spirits about everything, and certainly supports and is cooperating with everything that is going on," Pierson said of Dugard.
A trial date was not immediately set. Garrido is due back in court Feb. 28 to enter a plea.
Pierson would not say if Dugard wants to testify at the trial or whether his office is likely to offer a plea deal that would avert the need for one.