TAMPA, FL (CBS) - Two days after Herman Cain accused Massachusetts Senator Scott Brown of being "Republican only in name," Brown is responding, and other well-known Republicans are coming to his defense.

"I've never been a real big party guy," Brown told WBZ-TV Political Editor Jon Keller. Brown also suggested that Cain doesn't know what he's talking about.

"Herman Cain doesn't live in Mass," he said.

Brown was careful to quell any potential backlash back at home that he might feel from the convention's rhetoric, pointing out his appreciation for speeches by Ann Romney, Condoleezza Rice, Sen. John McCain, and others.

"They hit a right tone but didn't cross the line," he said.

Cain aside, most Republicans don't seem to begrudge Brown for his distance.

"What we want is to make sure those of us fortunate enough to serve in public office are focused on the people's issues and not on the politics," Bristol County Sheriff Tom Hodgson said. "I think that's what Scott Brown's all about."

Moderate former New York City Mayor Rudy Giuliani also praised Brown's position.

"I hope the people of Mass. understand how independent he is how unusual that is in American politics, what a great senator he's been for them because he's willing to vote his conscience; and how they shouldn't replace him with a pure ideologue," Giuliani said.

Brown also added some harsh words for opponent Elizabeth Warren, seemingly referencing a new round of Warren campaign ads targeting national Republicans.

"Anyone who's trying to paint me as an ultra-right conservative, it's Professor Warren and the Democratic Party trying to nationalize the race," Brown said. "My message to Professor Warren is, 'I'm right here.' She's running against Scott Brown, not Mitt Romney or Paul Ryan. And I'll be back tomorrow getting Mickey Ward's endorsement. She's welcome to come up to Lowell, if she can find it."