Abbas stopped short of urging an uprising, but he appeared to be trying to capitalize on growing discontent with Hamas in the coastal territory.
Provoking the tough talk was in Gaza City on Monday, killing eight civilians and wounding dozens in Fatah's strongest show of strength against the Islamists since the Gaza takeover.
"We have to bring down this bunch, which took over Gaza with armed force, and is abusing the sufferings and pains of our people," Abbas said in a speech.
An aide said the Palestinian leader, who set up a Western-backed government following Gaza's fall to Hamas, wasn't calling for a popular revolt against the militant group, rather reacting passionately to the bloodshed in Gaza City.
Despite Abbas' tough words, he has little influence in Gaza. Hamas, by contrast, has taken new steps to cement its grip on the territory since the rally, and announcing media restrictions and plans to limit public gatherings.
Hamas condemned Abbas' comments and accused him of selling out the Palestinian people in peace talks with Israel. A formal relaunching of negotiations is to take place later this month at a U.S.-sponsored summit that is also meant to bolster Abbas in his struggle with Hamas.
Deposed Prime Minister Ismail Haniyeh of Hamas, whose government rules Gaza, rejected Abbas' call. "Hamas is hard to break, and it is beyond being wiped out or eliminated," he said in a televised speech after sundown Thursday. "We are people whom without us things can't be finalized."
Haniyeh also said there would be an "honest, fair and transparent" committee to investigate the events at the rally. He said Hamas would free all the detainees who were not directly involved in inciting violence against his police.
He said a reconciliation committee has begun work in Gaza. Abbas has refused to talk to Hamas since its takeover of Gaza.
Five months of Hamas rule and crushing international sanctions have taken a heavy toll on already impoverished Gaza, and the hardship helped drive the Fatah rally, which drew about 250,000 people on Monday. Still, the heavily armed, Iranian-backed Hamas remains entrenched.
Hamas continues to bring in millions in cash through smuggling tunnels under the Egypt-Gaza border, enough to pay salaries to 16,000 members of its security forces and government employees.
Gaza militants are also undeterred. According to the Israeli military, they have fired more than 1,200 rockets and mortar shells toward Israel since June.
In northern Gaza on Thursday, Israeli soldiers opened fire on a car carrying militants, killing two and wounding five, militants and a health official said. The Israeli army said ground forces struck a group of militants about to fire rockets toward Israeli towns. In all, at least six rockets and mortar shells exploded in Israel on Thursday.
Abbas' call for an uprising in Gaza coincides with his efforts to bridge differences with Israel ahead of the summit in Annapolis, Maryland.
Abbas said in his speech that his government was "working relentlessly" to make the gathering a "decisive-launching pad" for establishing a Palestinian state. But he demanded that Israel halt all settlement construction, release Palestinian prisoners and end its targeted killing of suspected Palestinian militants.
Abbas spoke on the 19th anniversary of the Palestinian declaration of independence at a meeting in Algeria.
The declaration has not brought about the establishment of a Palestinian state, but is regarded as significant because it implicitly recognized Israel's right to exist for the first time.