"I was doing a joke in my show and that shouldn't influence public opinion. I don't know if it should go that far," Buress said Sunday. "It's weird to me that a joke did that."
Buress said he didn't perform the joke with the idea it would have repercussions for Cosby. He stopped short of expressing regret.
Speaking earlier to a TV critics' group, Buress said he's moved beyond Cosby as a punchline and doesn't see his career as defined by it.
"I've been working for a while and I continue to work," Buress said, citing his voice-over performances for films including the upcoming "The Angry Birds Movie" and ongoing stand-up appearances. A comedy special for Netflix will be released Feb. 5.
"That's just one joke that ... people took and really ran with it," he said.
In October 2014, Buress called Cosby a rapist during a Philadelphia stand-up performance. It was captured on video and posted online, gaining wide exposure and renewing scrutiny of decades-old allegations of sexual assault lodged by a number of women.
The chorus of accusers grew, eclipsing Cosby's public image as a charming, family friendly comedian and an African-American groundbreaker with his role in the 1960s in "I Spy" and his 1984-92 NBC TV sitcom "The Cosby Show."
Cosby had previously come under fire from some in the African-American community for critical remarks about the conduct of poor blacks.
Buress declined Sunday to discuss what prompted making Cosby a comedic target.
Cosby repeatedly has denied the sexual-misconduct claims, with his attorneys noting he had never been charged. That changed on Dec. 31, when he was arrested on felony assault charges in suburban Philadelphia stemming from a 2004 encounter with a former Temple University employee.
The decision represented an about-face by the district attorney's office, which under a previous leader declined to charge Cosby in 2005 when Andrea Constand first told police that the comic put his hands down her pants.
Responding last month to the charges, Cosby's attorney Monique Pressley accused the new prosecutor of "playing political football" with Cosby's life.
Buress, who appeared on the panel with John Mulaney and Patton Oswalt, drew support from his fellow comedians.
"Many, many people knew him as an excellent standup before that," Mulaney said.
Added Oswalt: "Hannibal was doing fine without that (the Cosby controversy). He'd be sitting here now if that had not happened. ... That's just incidental to where he was going anyway."