President Traian Basescu said it was Romania's duty to "recognize the genocide during World War II" and to honor the victims.
Basescu was joined by Holocaust survivors, both Jewish and Gypsies, and other leaders during the unveiling the euro5 million (US$7.35 million) marble and concrete tomblike monument.
Romania today has only 6,000 Jews. The country's role in the Holocaust and the deportation of Jews were ignored by the Communists and minimized by subsequent governments after communism collapsed in 1989.
"This monument is full of symbolism. Hundreds of thousands were killed who would have contributed to the cultural and economic prosperity of Romania," said Rabbi Menachem HaCohen, Romania's chief rabbi.
Romanian authorities set up the Elie Wiesel International Commission on the Holocaust in Romania in 2003 after one ministry in the Social Democratic government denied there had been a Holocaust in Romania during World War II.
Some Romanians admire pro-Nazi Marshal Ion Antonescu, whose administration was responsible for the deaths. They see him as a hero who fought against the Soviet Union to recover Romanian territories.
"It is important that Romania acknowledges its past. It's not an easy past," said Radu Ioanid who has written several books on the Romanian Holocaust. "There are still people opposing this," he said.
Historians have documented several pogroms in Romania, including one in June 1941 in the northeastern city of Iasi, where up to 12,000 people are believed to have died as Romanian and German soldiers swept from house to house, killing Jews. Those who did not die were systematically beaten, put in cattle wagons in stifling heat and taken to a small town. Of the 120 people on the train, just 24 survived.
Other Romanian Jews were deported from Transylvania by Hungarian fascists to Nazi concentration camps. Part of Transylvania was controlled by Hungary during World War II.
Nobel Peace prize laureate Elie Wiesel, a Holocaust survivor who was deported to Auschwitz, called on Romanians not to forget their past.
"Passer-by, stop and remember," he said in a message read out. "In this wonderful country and in Trans-Dniester under the criminal regime of Antonescu, hundreds of thousands of Jews were persecuted and assassinated in bloody pogroms."
Critics say that Romanians are beginning to learn about the Holocaust.
"Romania has made great strides in recognizing the past, especially the government educational institutions," said Michelle Kelso, director of the Association for Dialogue and Civic Education. "However to win the hearts and the minds of people, there needs to be more education campaigns especially on the lessons of the Holocaust."
Director of the Holocaust Memorial Museum in Washington Sarah Broomfield called the unveiling of the monument "extraordinary."
"To watch a young democracy dealing with history that was ignored by the Communists is admirable," she said. "Now they can write their own history."