Riordan, the wealthy former Los Angeles mayor known for his support of public schools, startled even friends last week with the comments at a promotional event for summer reading at the Santa Barbara library.
Schwarzenegger, his longtime friend, called Riordan's statement to the girl "unacceptable in any context" but gave no hint his job was in danger.
But Alice Huffman, president of the California chapter of the National Association for the Advancement of Colored People, said Thursday that Riordan "is not suitable to lead education in our state" and should be removed.
The conversation, videotaped by KEYT-TV, took place July 1. The girl, 6-year-old Isis D'Luciano, asked Riordan if he knew her name meant "Egyptian goddess."
Riordan replied, "It means stupid dirty girl."
After nervous laughter in the room, the girl again told Riordan the meaning of her name.
"Hey, that's nifty," he said.
A day later, Riordan issued a statement that said he "teased" the girl. "I immediately apologized to her, and I want to do so again for the misunderstanding," Riordan said.
The governor's chief spokesman, Rob Stutzman, said Thursday Riordan had not been asked to resign and, "That's the end of the issue, as far as we are concerned."
The girl's mother, Trinity Lila of Goleta, said her daughter was fine, and she considered the issue over.
"Obviously it hurt her feelings, but she didn't take it personally. She knew he was wrong and she let it go," Lila said. "I'm not going to sue them for therapy bills."
"He's already apologized repeatedly," Lila added. Although Riordan's office has tried to contact her, "I don't see what else is to be done."
But the issue has persisted. In an editorial Thursday, The Sacramento Bee said California "shouldn't have an education secretary who makes offensive, damaging remarks to young children for no apparent reason."
"He's the governor's top person on education, which means we would expect him to have some love and respect for children," Huffman said. "I think he is the wrong man for the job. There is no way for him to explain this away."
Riordan, a venture capitalist who started a foundation supporting literacy, has a reputation for awkward — some might say insensitive — remarks and behavior.
As mayor, he once greeted hunger strikers outside his office eating a hamburger. In a speech to school administrators earlier this year, he told a story about a nun physically disciplining a student that startled some in the audience.
"There wasn't an uproar, but some people felt the comment was inappropriate," said Joseph Jones, assistant executive director of the Association of California School Administrators.
Democratic state Assemblyman Mervyn Dymally, who had scheduled a protest by civil rights organizations, canceled the demonstration after an apparent mix-up over the girl's racial background.
Dymally was quoted in the San Jose Mercury News Thursday saying the child was "a little African-American girl. Would he (Riordan) have done that to a white girl?"
The girl is white, with blond hair.
Dymally did not return telephone calls. His office issued a statement Wednesday calling Riordan's remarks to the girl "outrageous and irresponsible," then issued another statement Thursday saying, "To err is human; to forgive is divine."
"Race is not a factor in this issue," Dymally said in Thursday's statement, adding that Riordan had apologized a second time. "It is time for us to move on."