Matthew J. Hoffman even sat nearby last week as police recovered the missing family's truck in central Ohio, Knox County Sheriff David Barber said.
Now, investigators acknowledge hope is dimming that the girl's mother, brother and family friend will be found alive nearly a week after they were reported missing.
"You have to be realistic that there's a possibility that these folks are dead," Barber said Monday.
Police on Sunday rescued 13-year-old Sarah Maynard from the basement of Hoffman's home, then launched a search of a nearby lake for Maynard's mother, 32-year-old Tina Herrmann; the woman's 10-year-old son, Kody Maynard; and her 41-year-old friend, Stephanie Sprang.
All four vanished last Wednesday from a Howard, Ohio, home that police say was splattered with blood.
Sprang's father, Steve Thompson, told CBS' "The Early Show" Tuesday morning that he couldn't take the warnings from the police onboard.
"Well, at this point, we're not listening to them. I'm sorry," an emotional Thompson told "Early Show" co-anchor Harry Smith. "As far as we know, they are alive and we will find them eventually."
"Maybe they've been tied up someplace and couldn't get free, and now they are," Thompson said. "That would make it all right. That would make me happy."
CBS News correspondent Dean Reynolds reports that neighbors gathered Monday evening for a vigil, just up the road from Hoffman's home, also hoping that the horror story unfolding in their town might still have a happy ending.
"I know that the time frame has been long between when they disappeared, but we have hope that we will see our loved ones soon," Thompson, told WBNS-TV in Columbus on Monday.
Hoffman, who's been charged with one count of kidnapping, is to appear via video from jail for a bond hearing Tuesday. The sheriff said he does not yet have a lawyer.
It wasn't clear how well Hoffman knew the four missing people, but the sheriff suggested that the defendant had been watching them.
"They knew Hoffman or Hoffman made himself known to them; he acquainted himself with the family whether they knew he was acquainting himself with them or not," Barber said.
Authorities first questioned Hoffman on Thursday, the day after Herrmann failed to show up for work at Dairy Queen. Police found him sitting in his car near a bike trail opposite property owned by Kenyon College, near where Herrmann's pickup truck was found, Barber said.
The sheriff didn't say what later led investigators to Hoffman's two-story, tan-sided house in Mount Vernon, where authorities spent Monday scouring bike paths and riverbanks. A search team pulled a car and an SUV from a lake near Hoffman's home, but investigators say they're not likely related to the disappearances.
"Our hearts were dropping every time they pulled something out, but now that we know that they were not related to this case, we're happy," Thompson told WBNS, with a tear rolling down his left cheek.
Authorities were to continue the search on Tuesday.
Hoffman's mother and stepfather live less than a mile away from Herrmann's home in a lakeside community north of Columbus. Hoffman last lived there two years ago, his mother said, before declining to comment further.
Hoffman was sentenced to eight years in prison in Colorado in 2001 for arson and other charges. Authorities allowed him to move to Ohio in 2007 after he was released on parole, which ended about a month ago. He had paid about $4,800 toward $2.06 million in restitution, Colorado court system spokesman Jon Sarche said.
In Ohio, the sheriff declined to comment on whether Sarah was assaulted or the details of her capture. She was released from a hospital and was staying with relatives.
"She is a very brave little girl," Barber said. "Under the circumstances, a 13-year-old girl being held captive for four days by a total stranger ... I would call her the epitome of bravery."
Two blocks from Hoffman's home in Mount Vernon, about 200 community members prayed and lit candles at a church vigil for the missing people.
"I'm expecting the worst but hoping for the best," said Jacki Mace, a manager at Sunset Lane Tanning, where Herrmann and Sprang were regular customers.
Mace, 20, said Herrmann was last in the salon Wednesday morning. She called Herrmann "a sweetheart."
"If you were feeling down that day she'd do anything to make you smile, pick you up, tell you a funny story. She was a delight," Mace said. "And Stephanie was the same. She was just a sweetie. They were so nice. Never had any problems with them."