Harmon and friends hold a charity baseball game every year, with the money going toward everything from nutrition counseling to new buildings.
"If we can lend three days of our time and impact this place, like, you know, obviously we have," Harmon said.
"You like that solid wall," Smith said. "It's the carpenter in you."
"Yeah, I like that it's there. I like that it'll remain there," he replied.
But Mark Harmon's life hasn't always been this serious. In January 1986 he was named People Magazine's "Sexiest Man Alive."
"Do you think you'll ever live down being the 'sexiest man alive'?" asked Smith.
"I never even accepted living it up! " he replied. "To be honest with you, I don't know who in their right mind could ever take that seriously. But believe me, there are guys who do."
Not long after he was named "sexiest man" he met and married his match: Pam Dawber, "Mindy" in the series "Mork & Mindy" with Robin Williams.
After the first of their two children was born, Harmon made a conscious decision to stick closer to home.
"I left to do a movie in New Guinea," Harmon said. "And our oldest had just taken his first steps the day I left. And I came back three months later and he got out of the car by himself and walked up the sidewalk and grabbed onto my leg at the airport.
"And I turned to my wife and I said, 'Hate to tell you, but I'm not gonna be doing new movies in New Guinea for the rest of my life to be able to afford the house we're in, and miss all this."
Despite his being a father (or maybe because of it), he once did something you usually only see in the movies.
On a quiet night in January 1996, two teens crashed their car near his L.A. home. One kid crawled free, the other was trapped in the car, and burning alive.
Harmon broke the car's glass with a sledgehammer, and pulled the trapped boy out of what was by then an inferno.
"When he came out of the car and he was full fire on his back," Harmon recalled. "And we got that done, put him down and got on top of him, smothered that."
" 'Stop, drop and roll' kind of thing?" asked Smith.
"Yeah, or being a lifeguard, which I was. And when I broke the window to get to him it fed oxygen -- the fire went from a two to a seven like that.
"There was no extra time for any of it. But I had endless time. I had nothing but time. I grabbed firm once and caught his belt. Upside down, blind in the car. He came out on his back in one push. The fire had burned through the seat belt, okay, so if the seat belt is hooked he's in there, I can't get him out. . . . You just react. That's all you do. If you think about it you maybe respond differently."
"You can't even take credit for that," said Smith.
"But there's moments, just moments that make the difference between the story we're talking about here and you saying, 'You can't take credit for it.' I won't take credit for it because if the car blows up and I'm there next to the car, then you're talking about two young boys that don't have a father and you'd be doing this interview with my wife and talking about how stupid it was. Right?"