The lawsuit, filed by Gay and Lesbian Advocates & Defenders, challenges a September ruling by Attorney General Tom Reilly that the referendum effort is legal.
That ruling allowed backers of a proposed amendment to the state Constitution to ban same-sex marriage to begin collecting signatures. They gathered more than 120,000 certified signatures — well above the 65,000 needed to get the measure on the 2008 ballot.
Gary Buseck, legal director of GLAD, said the state Constitution specifically bars any citizen-initiated amendment that "relates to the reversal of a judicial decision." Only the Legislature can propose that kind of amendment, he said.
A landmark 2003 court decision legalized same-sex marriage in Massachusetts.
Reilly should have blocked the question from going forward on those grounds, Buseck said.
A spokesman for Reilly did not immediately return a call seeking comment.
Kris Mineau, president of the Massachusetts Family Institute, one of the prime backers of the amendment, said he had been expecting the lawsuit but is confident the court will reject the argument.
"We feel confident that they will not succeed," Mineau said.
He said the proposed amendment does not specifically seek to reverse the Supreme Judicial Court's decision in the Goodridge case, instead seeking to clarify the legal definition of marriage so that it only applies to the union between a man and a woman.
The GLAD lawsuit was filed with a single justice of the Supreme Judicial Court on Tuesday and could be passed along to the full court.