Initial exit polls had forecast a slight edge for Fatah, with Hamas coming in a strong second. The polls predicted that neither party would have a majority and would have to rely on smaller parties to form a coalition.
Palestinian pollsters were at a loss to explain the discrepancy between the exit polls. Many voters said they had been afraid to admit to pollsters they had supported Hamas, fearing retribution.
"It's almost like the 1948 U.S. election when supposedly first Dewey won and then Truman," Brookings Institution Senior Fellow Michael O'Hanlon said on CBS News' The Early Show.
A Hamas-only government, without Fatah as a moderating force, is sure to throw Mideast peacemaking into turmoil. The Islamic militants, who carried out dozens of suicide bombings and seek Israel's destruction, have said they oppose peace talks and will not disarm.
"If we look for a silver lining we can see it did not really win because of its terrorism per se," but for the welfare services it has provided for Palestinians, O'Hanlon told Early Show co-anchor Harry Smith. There was also "frustration with the incumbent regime has been so great because it hasn't really delivered."
Ordinary Israelis consider it a bad sign, reports CBS News correspondent Robert Berger.
U.S. election monitors led by former president Jimmy Carter said Thursday the parliamentary elections, the first for Palestinians in 10 years, went well.
"Election day was generally peaceful and the elections thus far appear to be well-administered under the difficult circumstances of ongoing conflict and occupation," said a statement.
The Central Election Commission said the vote count had not been completed. An an official announcement was expected shortly.
Earlier Thursday, top Hamas leader Khaled Mashaal told Abbas his group is ready for a political partnership.
However, Fatah does not want to join a Hamas government, said Fatah legislator Saeb Erekat. "We will be a loyal opposition and rebuild the party," Erekat said, after meeting with Abbas.
Abbas, who favors peace talks with Israel, has said he would resign if he could no longer pursue his agenda.
Israel and the United States have said they would not deal with a government led by Hamas, which has carried out dozens of suicide bombings and which they consider a terrorist group.
"Hamas is a terrorist group. It's on a terrorist list for a reason," President Bush said last May.
And only Wednesday, he said the U.S. won't deal with Hamas unless it renounces violence. The U.S. had been calling for Palestinian democracy. Following the vote, it now faces major new policy decisions, reports CBS News correspondent Peter Maer.
Secretary of State Condoleezza Rice called Palestinian leader Mahmoud Abbas to praise Palestinian democracy and assert that the United States supports him and his policies, Abbas' office said Thursday.
"She asserted to him that U.S. administration will continue supporting the elected president and his policies," said Nabil Abu Rdeneh, an Abbas adviser.
Acknowledging the Hamas victory, Palestinian Prime Minister Ahmed Qureia and his Cabinet ministers resigned Thursday.
"This is the choice of the people. It should be respected," Qureia said. The Cabinet remained in office in a caretaker capacity.
Under the law, Abbas must ask the largest party in the new parliament — presumably Hamas — to form the next government. Abbas was elected separately a year ago and remains president.
Hamas capitalized on widespread discontent with Fatah's corruption and ineffectiveness. Much of its campaign focused on internal Palestinian issues, while playing down the conflict with Israel.
There was no immediate official Israeli reaction. Acting Prime Minister Ehud Olmert said Wednesday, before Hamas claimed victory, that Israel cannot trust a Palestinian leadership in which the Islamic group has a role.
"Israel can't accept a situation in which Hamas, in its present form as a terror group calling for the destruction of Israel, will be part of the Palestinian Authority without disarming," Olmert said in a statement issued by his office.
"The peace process wasn't going anywhere anyway," said Brookings' O'Hanlon.
Hamas' exiled supreme leader, Khaled Mashaal, called Abbas from Syria to discuss the results. "He stressed Hamas insists on a partnership with all the Palestinian factions, especially our brothers in Fatah," Hamas said on its Web site.
Before the election, Hamas had suggested it would be content as a junior partner in the next government, thus avoiding a decision on its relationship with Israel. Throughout the campaign, leaders sent mixed signals, hinting they could be open to some sort of accommodation with Israel. Its apparent victory will now force it to take a clearer position on key issues, including whether to abandon its violent ideology.
Fatah's official position wasn't immediately clear. Officials appeared to be in shock, turning off their phones and avoiding reporters. There was no reaction from Abbas.
The election marked the first time Hamas has contested a legislative vote, and leading the Cabinet could give it significant powers. The Cabinet holds wide control over security forces, finance and other government functions, though Abbas has retained power mainly through tradition and political leverage.
Abbas, in his role as leader of the Palestine Liberation Organization, would still remain in charge of negotiations with Israel.
Palestinian legislator Hanan Ashrawi, who apparently was re-elected on a moderate platform, said the Hamas victory was a dramatic turning point. She said she is concerned the militants will now impose their fundamentalist social agenda and lead the Palestinians into international isolation.
She said Fatah's corruption, Israel's tough measures and international indifference to the plight of the Palestinians were to blame for Hamas' strong showing.
Washington miscalculated in pushing for the vote, as part of its pro-democracy campaign in the Arab world, she said. "The Americans insisted on having the election now, so they have to respect the results of the election, as we all do," she said.
Israel has repeatedly asked Abbas to force Hamas and other militant groups to disarm but Abbas has refused, warning such an act could cause civil war. Hamas has committed dozens of suicide bombings against Israel.
Turnout was heavy, with nearly 78 percent of 1.3 million eligible voters casting ballots. Polling stations were heavily guarded, and there were no reports of major violence.