Wearing an orange inmate jumpsuit with his hands and feet shackled, Rudolph had no comment as he was led into the county jail in Birmingham.
Rudolph, 36, appeared in federal court earlier Monday in Asheville, N.C., acknowledging his identity but entering no plea. He is charged in four bombings, three in Atlanta and one in Alabama, and could get the death penalty.
Afterward his attorney, Sean Devereux, said Rudolph was innocent and insisted his client is not the extremist many Americans believe.
"He's a reflective individual. He's been thinking a lot. He's been through a lot," Devereux said.
He also said Rudolph told police the location of his campsite in western North Carolina, where he is believed to have hidden for the past five years before he was captured Saturday while scavenging for food behind a supermarket in Murphy, N.C.
Attorney General John Ashcroft said in Washington that Rudolph will be tried first in Birmingham, where the New Woman All Women Health Care clinic was bombed in 1998, and then in Atlanta, site of the 1996 Olympic bombing and two other blasts linked to Rudolph.
"Our approach is designed to provide the best opportunity to bring justice to all of the victims of the bombings," Ashcroft said in a statement.
"I think the Feds want to win right out of the box," says CBSNews.com Legal Analyst Andrew Cohen. "I think they want to go to Birmingham because they think their case there is strong. There was an off-duty cop killed, and an eyewitness that can link Rudolph to the crime that no one in Georgia can."
Evidence gathered after the Birmingham bombing on Jan. 29, 1998, led to Rudolph being identified as the lone suspect in all the blasts, which killed two people and injured more than 150.
A witness saw a man believed to be Rudolph leaving the scene of the Birmingham bombing, which killed an off-duty police officer and critically injured a clinic nurse. A truck registered to Rudolph was spotted moments later.
In the days after the Birmingham bombing, agents searching a storage locker rented by Rudolph found nails like those used to bomb the clinic and an Atlanta building that housed an abortion clinic.
Authorities said the bombs had similarities that also linked them to the Olympic blast, which killed one person and injured dozens, and the bombing of a lesbian bar in Atlanta.
Rudolph had been on the FBI's 10 Most Wanted list, and the government had offered a $1 million reward for information leading to his capture.
Authorities want to know if the former fugitive got any help from local residents. In the mountains where he hid out for five years, there appeared to be support for Rudolph.
CBS News Correspondent Jim Stewart reports they're still selling bumper stickers near the Natahalia Forest that say "Run Rudolph Run" and "Eric Rudolph: North Carolina Hide & Seek Champ."
Federal lawmen were not amused. Former ATF agent Jack Killorin spent five years looking for the man.
"This is a person whose crimes differ only in a matter of degree from the crime of Sept. 11. These were deliberate attempts at mass murder," Killorin said.
Nor do lawmen forget that they were the likely targets. A 911 call before the Centennial Park bombing was just to sucker cops into the killing zone, they think. The next attack at an abortion clinic was actually two bombs – one to summon police, the next to kill them.
As for some "Army of God" letters stating a philosophy, lawmen don't buy that either.
Rudolph bombed a gay nightclub, yet he has a beloved gay brother. He professes to hate abortion, yet his mother says he never mentioned it. He denies the Holocaust, but can't explain why. He flirted with the white supremacist Christian Identity movement, but never joined.
Rudolph did join the Army, and took copious notes during explosives training. But then he got bored, agents say, and smoked so much dope he drew an early discharge. Back home, he was known for just hanging out in the woods and renting video movies nonstop.
"Mr. Rudolph seems a person who lived kind of hand to mouth existence as a carpenter and it's hard to find a job that he held and held successfully," says Killorin.
About the only two things Rudolph ever excelled at, lawmen say, were growing marijuana and building bombs. The dope was potent and the bombs displayed a learning curve, cops say. The last one went off by remote control.