Al-Jaafari won 64 votes, one more than Vice President Adil Abdul-Mahdi, officials said. There were two abstentions.
More than 100 lawmakers from the Shiite coalition, the United Iraqi Alliance, gathered to vote.
The choice of the umbrella Shiite alliance is assured of becoming prime minister because Shiites won the most parliament seats in the Dec. 15 national elections.
Shiite lawyers cast their votes at the heavily guarded home of Abdul-Aziz al-Hakim, the head of Abdul-Mahdi's party. Al-Jaafari's supporters gathered in the compound cheered when word of the outcome emerged from the closed door meeting room.
Al-Jaafari, a physician, is a member of the Dawa Party and spent years in exile in Iran and Britain before returning to his homeland after the U.S.-led coalition ousted Saddam Hussein in 2003.
His government, which took office in April 2005, had been widely criticized for failing to improve the country's crumbling infrastructure or deal effectively with the Sunni-led insurgency.
Meanwhile, the chief lawyer representing Saddam Hussein said Sunday he was wrong in reporting that the former Iraqi leader and seven co-defendants would begin a hunger strike to protest the "illegality" of the court hearing their case.
Khalil al-Dulaimi who had initially reported that the hunger strike would begin Monday, saying he received the information through sources at the detention center where Saddam and the other defendants were being held.
"I checked and I was told that the sources were not credible and that there will be no hunger strike on Monday," al-Dulaimi told The Associated Press. He declined to provide other details, saying he will issue a written statement later.
In other developments:
The dominant Shiite coalition, the United Iraqi Alliance, won 128 seats. The Kurdish Coalition of parties led by President Jalal Talabani and Kurdish leader Massoud Barzani were the other big winners, claiming 53 seats.
He said two Sunni Arab blocs, the Iraqi Accordance Front and the Iraqi Front for National Dialogue, received 44 and 11 seats respectively. Sunni Arabs now have three times more seats than in the outgoing parliament.
Sunni Arabs form the backbone of the raging insurgency, and bringing them into the government is seen as a way to reduce the violence.
Under Iraq's new constitution, Talabani not only must convene the new assembly, he must also name a new prime minister in the next 15 days. Parliament then has 30 days to elect a new national president.
If the Iraqis take the maximum time allowed for each step, it would be May before a government is in place. Vice President Adil Abdul-Mahdi, a Shiite widely mentioned as possible prime minister, has said he expected to finish the talks by mid-March.