"A lot is being made about us and what we do, how we judge, who's on, who's not on, what's happening, all of that," he said in an interview Tuesday.
"It's really about those contestants, really. They're going to go on because it's them taking the rocket ship to try and get their fame started ... I mean, so much is being made about us or whatever."
That's because the changing faces of the judges have provided the biggest news.
New "Idol" Anthem: "Pants on the Ground"
Cowell Called Irreplaceable
Will "Idol" Hang on as No. 1 Series?
Photos: "American Idol" Season Eight
Photos: The "Idol" Finale
Photos: Ellen DeGeneres
Photos: Simon Cowell
Simon Cowell recently announced he's leaving "Idol" for the U.K.-based competition series, "The X Factor," and Paula Abdul left after her contract deal fell through.
Jackson, along with Kara DioGuardi, Cowell and various guest judges, kicked off the ninth season of "Idol" last week. Ellen DeGeneres will begin her stint on Feb. 9, when the contestants who received the golden ticket compete in Hollywood.
Jackson, 53, laughs when he's asked about the length of his contract with the Fox Television series: "It's got some time left, yes," he acknowledges.
And ask him if he would work as a judge on "The X Factor" and he'll laugh again.
"You never know what's going to happen, you just never know in life," he added.
He says finding a judge to replace Cowell will be "tough" and "really difficult" because "there's nobody quite like him."
He later jokes about Cowell's possible replacements. "Maybe Conan? Maybe Leno? I don't know," he said.
Throughout the season nine auditions, Jackson says the girls outshine the boys - and he's hoping a female takes the top prize.
"In my mind, the biggest winners ever on 'American Idol' have been girls. I mean that have gone on to huge success with Carrie Underwood, with Kelly Clarkson, these are your big, huge, giant success stories," he said.
"We had two years of boys," he said of season seven and eight champions, David Cook and Kris Allen, respectively. "C'mon, we need a girl!"
Jackson may be rooting for the ladies, but there's one male contestant that he - nor America - can't get enough of.
"General" Larry Platt, the 63-year-old civil rights veteran who auditioned in Atlanta, has become an Internet sensation with his original song, "Pants On the Ground."
"I'm so happy he's getting his day and his moment and I even like the message in his mind, his wisdom, that he's trying to impart on the young hip-hop community," Jackson said.
The next question is whether "Pants On the Ground" could top the charts? Jackson thinks so.
"Dude, let's go. Let's get the remix, man! What? Give me Jay-Z, put Jay-Z on it," Jackson said laughing.