Several current and former players say Sterling is not fit to own a team. The NBA is under pressure to take action and has called a news conference for Tuesday.
Sterling was tight-lipped as he left a Los Angeles restaurant Sunday night, but his wife, Rochelle, loudly defended him.
"Not true," she says in a Splash News video, when reporters asked whether Sterling was a racist.
It was a change for Rochelle Sterling, who had initially joined the chorus of people calling her husband "despicable" after website TMZ released an audio recording alleged to be of Sterling talking with his former girlfriend, V. Stiviano.
"It bothers me a lot that you want to broadcast that you're associating with black people," the person identified as Sterling says in the recording.
Stiviano attempted to avoid cameras Monday but says she recorded an argument with Sterling after he got angry when she posted her picture with Magic Johnson on the social media site Instagram.
"Why are you taking pictures with minorities? Why? You can sleep with them, you can bring them in, you can do whatever you want. The little I ask you is not to promote it on that and not to bring them to my games," the man on the recording says.
The Clippers wore their practice shirts inside out, hiding the team's logo, in a silent protest before their playoff game Sunday. Coach Doc Rivers says he refused a call from his boss, Sterling. And many NBA greats, including Johnson, Michael Jordan and Kobe Bryant, are voicing strong disapproval.
"We can't have people like that representing the NBA," Hall of Famer Kareem Abdul-Jabbar, who once worked for Sterling and the Clippers, told CBS News.
Abdul-Jabbar said the voice on the recording sounded like Sterling.
"It sounded like him to me, it really did," he said. "I wasn't surprised, because I just was aware of his track record, the discrimination suits against minority people trying to rent some of his properties."
In 2009, Sterling paid a record $2.7 million to settle a housing lawsuit claiming he discriminated against African-American and Hispanic renters.
On Monday, the NAACP rescinded a lifetime achievement award it was planning to give to Sterling next month.
"His organization gave more money to the minority community than others," Los Angeles chapter President Leon Jenkins said, explaining why Sterling was initially picked. Jenkins said Monday that the NAACP would return all donations from Sterling.
So far, more than a dozen sponsors have cut ties with the Clippers, including CarMax, Corona and Mercedes Benz. The NBA has hired an audio forensics expert to listen to the voice on the recording and try to determine if it is that of Sterling.