Experts in the region called the arrests a major blow to one of the most brutal rebel groups in Africa.
German federal police arrested Ignace Murwanashyaka, the 46-year-old chairman of the Democratic Forces for the Liberation of Rwanda militia, and his 48-year-old deputy, Straton Musoni, in Karlsruhe, prosecutors said in a statement.
The pair are "suspected of carrying out crimes against humanity and war crimes" as members of the group, which is known by its French acronym FDLR.
Founded in the Congolese town of Lumumbashi in 2000, the FDLR is a Hutu extremist group made up of Hutu refugees from Rwanda who took cover across the border in Congo after the 1994 genocide of half a million Tutsis in Rwanda.
The United Nations has been putting pressure on Germany for years to arrest Murwanashyaka as well as Musoni, one of two vice presidents of the group. Murwanashyaka has served as the leader of the FDLR since 2001, prosecutors said.
"As part of this armed conflict, the FDLR militias are believed to have killed several hundred civilians, raped numerous women, plundered and burned countless villages, forcing villagers from their homes and recruiting numerous children as soldiers," the statement said.
The arrest is important because Murwanashyaka is the highest-ranking leader of FDLR, said Gregory Alex, a veteran U.N. official who now heads a United Nations unit in eastern Congo dedicated to demobilizing FDLR and other combatant.
"And this is a sign that the activities of the FDLR are being paid attention to by the international community. That they are seen as crimes against humanity," Alex said.
Julien Paluku, the governor of North Kivu province in eastern Congo, hailed Murwanashyaka's arrest.
"I think the arrest of Mr. Murwanashyaka will have a psychological impact on the morale of FDLR's militiamen who could be discouraged by this arrest," Paluku said. "Other leaders who will try to be speaking publicly for FDLR will fear that they too can be arrested like Mr. Murwanashyaka."
The FDLR is believed to number less than 5,000 men but they have become the excuse for numerous other equally brutal militias, which have cropped up in Congo over the years under the guise of protecting Congolese civilians. Experts say that dismantling the FDLR is key to solving the unending spiral of violence in Congo, a country the size of Western Europe that has suffered back-to-back wars since the end of the Rwandan genocide.
Gen. Paul Rwarwakabije, the former military chief of the FDLR who deserted several years ago and now works for the Rwandan government, said Murwanashyaka's arrest would significantly weaken the group.
"He was the head of the movement. This is an organization that has had its head cut off. Its brain has been taken away," said Rwarwakabije in a telephone interview from Rwanda.
The pair are to be brought before a judge later Tuesday.