Paul Shaffer recalls unforgettable "Late Show" moment

David Letterman will always be remembered for his groundbreaking comedy, but as "Late Show" musical director Paul Shaffer pointed out, a more serious moment may be his finest.

"Who will ever forget that he was the first man in late night to go back on the air after the 9/11 attack?" he said Friday on "CBS This Morning." "Everyone was looking toward him to see how he'd do it. And when he did it, he made it OK for everyone to come back."

Letterman returned to the airwaves after the Sept. 11 terrorist attacks in an Emmy Award-winning broadcast exalted by the New York Daily News as "one of the purest, most honest and important moments in TV history."

He delivered one of his most memorable lines in that show's monologue.

"If you didn't believe it before -- and it's easy to understand how you might have been skeptical on this point -- if you didn't believe it before, you can absolutely believe it now: New York City is the greatest city in the world," Letterman said.

Letterman's 33 years on the air as a late-night talk show host will come to an end Wednesday, May 20. He made the announcement on his show last year, but not before breaking the news to Shaffer.

"It was nutty. We were just about to go out and do our warm up. The band is playing, I'm about to go out, then Dave comes out to say 'Hello' to the audience just before we roll the tape and he says 'Just come here; come here for a second,'" Shaffer said. "And we went in a kind of an alcove, backstage in that old Ed Sullivan Theater and he said, 'I told them I'm retiring,' 'Really?' .... Everything felt different."

But for a man who has worked in every corner of the entertainment business, change is nothing new. The Ontario native has tickled the ivories for the original "Saturday Night Live" band, mastered acting on the big and small screen and performed with some of the world's top artists.

"Late Show" Music Director Paul Shaffer, talks with Dave about some of his greatest musical memories from the show, on the "Late Show with David Letterman," Wednesday May 13, 2015 Jeffrey R. Staab, CBS

Letterman has called him a musical genius.

"Well he exaggerates of course, but he has been so supportive of me," Shaffer said. "He has heard every single note that I and my band have heard over all these 33 years."

Shaffer said Letterman "never had anyone else in mind" for the "Late Night" role.

"He has just been the most incredible boss saying to me 'If you have anything, I don't care if it's in the monologue, I don't care if it's interviewing Julia Roberts, jump in at anytime.' Who has an open mic for 33 years? I had that. What could one say?," Shaffer said.

At 65, Shaffer has recorded more than 6,000 episodes with Letterman. He said the late-night legend never threw away a single show and maintained an "amazing level of perfection."

"It was a lot of fun every day and he kept it so real and so loose and so spontaneous," he said. "It was different every day and I never knew what he was going to do."

Life after Letterman won't be dull for Shaffer. He said he's going to keep playing the piano, and above all, listen to his wife Cathy's direction: "You gotta keep working."

"David let me do comedy, improvise, act. I want to do all those things, continue to do all those things," Shafer said. "Maybe a lovely a three-episode arc on 'CSI: Miami.'"