And in Washington, he's a fascination: at 51, a youngster in Senate terms, he's probably the only member who's a regular in triathlons and half ironmans. He may look a little like the actor Richard Gere, but his dismal childhood was more like David Copperfield's.
"How many moves did you make, how many different houses when you were growing up?" Stahl asked.
"Within the first 18 years, it was 17 different moves, 12 separate houses, and that doesn't count the times I ran away," he replied.
Back in his hometown of Wakefield, Mass., he tells us about how he suffered as a little boy from physical and psychological abuse at the hands of his mother's many husbands.
His first stepfather was Dan Sullivan.
"He was kind of a mean cuss," Stahl remarked.
"Yeah, he was a truck driver. Hot tempered. You know, I remember him being a pretty good drinker," Brown said.
In the book, Brown tells what happened when he woke Dan up from a hangover. He "plowed into (me) with those massive knuckles until I was shaking, sobbing, snot pouring out of my nose."
"And then he starts beating up your mother. There's an incredible scene in the book, you're 6 years old, and you jumped to your mother's defense?" Stahl asked.
"Yeah. I was in bed, and I heard her screaming. And I remember him hitting her...just pounding on her. And I remember just grabbing his leg, and just biting it right in the inside of his thigh," Brown remembered. "He was just pounding because I wouldn't let go. And was trying to shake me off, and everything."
But when Brown was 10, he experienced another kind of abuse at the hands of a camp counselor.
"You tell us that you were actually sexually abused. More than once," Stahl remarked.
"Fortunately nothing was ever fully consummated, so to speak. But it was, certainly back then, very traumatic," he said.
"You say it wasn't consummated. But you knew...he touched you," Stahl said.
"Yeah, he touched me," Brown said.
"And he made you touch him," Stahl said. "And then you thought you had escaped, and he came for you again. ...More than once."
"As predators do," Brown said. "He said, 'If you tell anybody I'll kill 'ya. You know, I will make sure that no one believes you.' And that's the biggest thing. When people find people like me at that young, vulnerable age, who are basically lost, the thing that they have over you is they make you believe that no one will believe you," Brown said.
He never reported it, and told no one, including his own mother.
"My mom will read about it for the first time. My wife (too)... I haven't told anybody. That's what happens when you're a victim. You're embarrassed. You're hurt," he said.