The veteran sportscaster is currently host of "Costas Now" on HBO.
King, 71, celebrated the 20th anniversary of his show last week by interviewing former presidents Bush and Clinton, Watergate sleuths Bob Woodward and Carl Bernstein and CBS' Dan Rather. Despite a swirl of change around him, he's still CNN's most popular personality.
But his May average of 970,000 viewers represented his worst month in four years, according to Nielsen Media Research. He regularly loses his time slot to Fox News Channel's "Hannity & Colmes."
CNN/U.S. President Jonathan Klein said the idea that Costas could be an eventual successor to King wasn't part of his thinking. Costas was available and Klein said he thought he'd be well-suited to the task.
"Larry King is far and away the superstar of CNN and if you list any of the areas we need to address at CNN, 9 o'clock Sunday through Thursday is not on the list," he said.
One of King's most frequent substitutes, Nancy Grace, has been doing well with her own legal-oriented prime-time talk show on CNN Headline News.
Costas, 53, has been a host for the Olympics and has covered the World Series, Super Bowl and National Basketball Association during his 25 years at NBC Sports. He also hosted the late-night talk show "Later with Bob Costas" for six years.
His first appearance for King will be on Sunday night.
Costas said he would not have agreed to the one-year deal without King's blessing. He said he was looking for the work because he has no on-air assignments for NBC Sports between the U.S. Open golf tournament and next winter's Olympics.
He also said he's not considering it any kind of tryout to be King's eventual successor.
"I'm not approaching it that way and Larry shows, to me, no sign of slowing down," he said.
By David Bauder