The agreement marks a breakthrough in the quest for justice 24 years after the Khmer Rouge lost power and six years after Cambodia first asked for U.N. assistance in holding a tribunal.
None of the Khmer Rouge leaders has ever faced trial for the atrocities committed during their 1975-79 rule, when nearly a quarter of the Cambodian population died.
Sok An, Cambodia's chief negotiator of the pact, and Hans Corell, the U.N. deputy secretary-general for legal affairs, signed the document at Chaktomouk conference hall, which the government has set as the venue for the internationally-assisted tribunal.
The agreement must still be ratified by Cambodia's legislature, and Corell and others have warned that it may still be a long time before the trials are convened.
Many senior Khmer Rouge figures live freely in the country, having surrendered to the government before the movement's collapse in 1998. The group's leader, Pol Pot, died the same year.
Pol Pot and other Khmer Rouge were educated in France but greatly influenced by the most radical aspects of China's communist revolution. They tried to purge their country of Western influences and base an economy on massive agricultural communes.
Many of those who died under the Khmer Rouge were victims of failed utopian plans that led to starvation and disease. Others were tortured and killed in purges.
The tribunal agreement has been opposed by several leading human rights groups, including Amnesty International, who said it will be set up in a way that makes it too vulnerable to political influence.
The Cambodian government and the United Nations began talks in 1997 on setting up an international tribunal in Cambodia, but struggled to agree on how much control foreigners would have over proceedings.
The United Nations was concerned about giving too much power to Cambodia's corrupt and politicized judiciary, while Cambodia feared infringement of its sovereignty.
U.N. negotiators abruptly pulled out of talks in 2002, claiming Cambodia was insincere in guaranteeing conditions for fair trials. Talks resumed last year after a gap of about a year.
Both sides finally concluded a draft agreement in March allowing joint teams of Cambodian and international prosecutors. Most of the judges will be Cambodians, but at least one foreign judge must support any tribunal ruling.
The Khmer Rouge came to power after Cambodia was destabilized by the war in neighboring Vietnam, especially Viet Cong and U.S. military incursions across the border, as well as massive American bombing conducted in secret.
They were driven from power in 1979 when Vietnam, Cambodia's long-time rival, invaded the country. Until the 1998 surrender, remnants of the force controlled remote areas in the country's north and west.