The Socialist Party spokesman on Wednesday joined the leader of the far right National Front party in questioning Mitterrand's fitness for his job.
Mitterrand, nephew of late President Francois Mitterrand, said it was a "disgrace" that his fellow leftists had joined in the criticism.
Mitterrand's 2005 book, "La mauvaise vie" or "The Bad Life," describes painful periods in his childhood and his homosexuality. One passage describes his "bad habit of paying for boys" in Thailand. Mitterrand later denied he was a pedophile, saying on France-3 television that he uses the term "boys" loosely.
Mitterrand was a television personality when the book came out but not in the government. President Nicolas Sarkozy named Mitterrand culture minister in June this year.
Concern about the book resurfaced in French political circles after Mitterrand's impassioned defense of Roman Polanski last week. The director was arrested in Switzerland on U.S. charges of having sex with a 13-year-old girl in 1977 in Los Angeles.
On Monday, National Front leader Marine Le Pen read excerpts of Mitterrand's book on French television and demanded his resignation.
On Wednesday, Socialist Party spokesman Benoit Hamon said he was shocked by the book's contents and questioned how France could continue fighting against sex tourism with Mitterrand in the government.