Rob Johnson, a spokesman for the Tennessee Department of Children's Services, said the children were taken from their mother, Maria Gurrolla, "purely for safety reasons," though he would not detail why the state deemed they were in danger.
"Our focus is on the children, and under the current situation right now, we think the safest thing to do is take the children into state custody," he said.
Gurrolla, 30, was stabbed in her home Tuesday, just four days after giving birth to Yair Anthony Carillo, who was snatched by the attacker.
Nashville police said the baby was found in good health Friday night at a home in Ardmore, Ala., about 80 miles south of Nashville near the Tennessee line.
Tennessee Bureau of Investigation Director Mark Gwyn said officials arrested Tammy Renee Silas, 39, at the home in Ardmore. Federal authorities formally charged her Saturday with kidnapping.
Silas' live-in boyfriend, Martin Rodriguez, said he was shocked by the arrest and didn't think she was capable of the crime.
Speaking through an interpreter, he said Silas told him she was adopting a baby from a cousin who had to go to jail, and was going to El Paso, Texas, to get the child. He said he picked Silas up from the Huntsville airport Tuesday and she had a newborn with her.
"She was acting normal around the baby and I didn't really see any difference, but I think she was happy," Rodriguez said from their one-story home, where a box of baby clothes for a boy overflowed in the dining room. "What woman isn't happy to carry a baby?"
Rodriguez said he met Silas, a contractor, when they both lived in Nashville. He said Silas is bilingual and was born in Tarrant County, Texas, where she had family. "The last thing that she said to me was, 'I am so sorry and I love you,"' Rodriguez said.
Earlier Saturday, officials said the newborn would stay with a foster family as authorities made arrangements for Gurrolla to be reunited with her son.
"This baby is a week old, and this child has spent half his life away from his family. I think it's time we reunite them," said My Harrison, a special agent with the FBI in Tennessee.
Johnson said officials made arrangements for Gurrolla to see her baby Saturday afternoon and hold him, and she brought her three other children - ages 3, 9 and 11 - with her. All four children were then taken into custody. Johnson, who said he could not discuss details of the situation for privacy reasons, said a judge would review the case next week to determine when the children can go home.
Joel Siskovic, an FBI special agent in the Memphis division, said he could not say why the children were put into state custody. "As of now, there's no indication that there's an ongoing threat to the family," he said.
Authorities said they had no word on a possible motive in the kidnapping. Police in Nashville did not know if Silas has a lawyer. The Morgan County Sheriff's office said Silas was picked up by U.S. Marshals on Saturday morning, though it was not known where she was being taken.
The infant's mother told police a heavyset white woman with blonde hair arrived at her home posing as an immigration agent and attacked her with a knife.
Gurrolla told investigators that at one point she heard the woman make a phone call and tell someone in Spanish words to the effect of "the job is done" and that the mother "was dying," said Siskovic, the FBI agent.
Siskovic said Silas took the victim's cell phone, which helped investigators locate Silas.
He would not comment further on the possibility that Silas was not working alone.
At a Wednesday news conference, Gurrolla told reporters she had never seen her attacker before.
Officials believe Silas followed Gurrolla and her baby from a local office of the Women, Infants and Children program and to a Walmart store. "I think it's clear that she was targeting people at that location," Siskovic said.
Investigators got a break when they found that a video camera in the Walmart parking lot had captured the license plate of the car seen following the mother and baby, according to the arrest warrant.
Cathy Nahirny, a senior analyst for infant abduction cases at the National Center for Missing & Exploited Children, said there have been at least two other recent cases where an abductor used a ploy similar to the one used in this case.
"We need to get the word out to our immigrant communities," Nahirny said. "Anybody that claims they are from federal law enforcement agencies, you have the right and you should ask for photo identification."
Abductions of infants by strangers are rare, with nine reported cases so far this year and five last year, according to the missing child center.
Nahirny said immigrant families have been targets of child abductions because of the assumption they will not tell police.
Gurrolla is Latina but her immigration status isn't clear. She was released from the hospital Thursday.