Zapatero also shifted Spanish foreign policy by pulling troops from Iraq in his first term, which he won three days after Islamic militants killed 191 people in a string of bombings against commuter trains.
Voters handed Zapatero his second term despite worries about a slumping economy, immigration and resurgent Basque separatists, blamed for gunning down a member of the prime minister's party on Friday - timing that recalled the March 11, 2004 Madrid attacks.
With 99.4 percent of the vote counted, Zapatero's Socialist party had 43.7 percent, versus 40.1 percent for the conservative Popular Party, according to the Interior Ministry.
In his next term, Zapatero's main task will be to reboot the once booming but now slowing economy, shaken by the sub-prime mortgage crisis in the U.S. and a cooling construction sector.
"The Spanish people have spoken clearly and decided to start a new era," Zapatero told euphoric supporters outside the party's headquarters in Madrid. "I will govern with a firm but open hand ... I will govern for all, but do so thinking most of all of those in need."
While Zapatero's Socialist party picked up seats in the lower house, it fell short of a majority and will have to form some sort of an alliance with smaller regional parties in order to govern.
The opposition conservatives conceded defeat, but took solace from the fact their party also picked up seats, with both parties gaining at the expense of smaller leftist and regional groups.
Zapatero paid tribute to the slain politician, Isaias Carrasco, saying he should have been celebrating the victory with his family. Carrasco's killing, blamed on the Basque group ETA, jolted Spaniards and prompted both parties to cancel final campaign appearances.
Some in Spain had predicted the killing might prompt a wave of sympathy and a boost at the polls for Zapatero's party, especially after the Socialist politician's 20-year-old daughter Sandra made an emotional appeal Saturday for people to defy ETA by turning out to vote en masse.
For Rajoy, his rival in Sunday's vote and in 2004, it was the second consecutive defeat, one likely to increase pressure on him to step down as party chief.
"I thought Rajoy would do better, he speaks with such conviction. I'm surprised he fell behind," said Jose Eguren, a 29-year-old security guard in the Basque city of Bilbao.
Many conservatives consider Zapatero's 2004 victory a fluke, and saw Sunday's vote as their chance to correct it. The prime minister's victory was seen as finally giving him a legitimacy that critics say he has lacked.
He won in 2004 amid a wave of voter outrage at the ruling conservatives, who blamed the attacks on ETA even as evidence of Islamic involvement mounted. The tactics were widely seen as a bid to deflect perceptions that the killings were al Qaeda's revenge for the government's deeply unpopular support of the U.S.-led invasion of Iraq.
Sunday's results showed the Socialists on their way to winning 169 seats in the 350-seat lower house, up from the current 164 but shy of the 176 seats needed for an outright majority. The Popular Party was also shown picking up seats, raising its total from 148 to 154.
Although he made a dramatic entrance with the withdrawal of troops from Iraq, Zapatero has since taken a back seat on the international stage. Some say his reticence to make a bigger impression has weakened Spain's standing abroad. Others believe he may tend to foreign affairs with more emphasis this time round.
The campaign was marked by acrimony, with Rajoy hammering Zapatero on everything from immigration to the economy.
In two televised debates between the men, Rajoy used a form of the word "liar" to describe Zapatero more than 30 times; he blamed Zapatero for not doing enough to spur the economy, which is cooling amid rising unemployment and an end to a boom in the construction sector.
Rajoy vowed to make immigrants sign a contract obliging them to respect Spanish customs and learn the language, a position Zapatero's party called xenophobic. The candidates also clashed on Zapatero's willingness to grant more self-rule to Spain's semiautonomous regions. Conservatives warn that will tear the nation apart.
Under the Socialists Spain became the third country to legalize gay marriage and thousands of same-sex couples have wed since the law took effect in July 2005, according to the Justice Ministry.
The government also pushed through laws including fast-track divorce and easier terms for medically assisted fertilization.
All of these issues have left Spain deeply polarized and these divisions will not go away soon, said Enrique Monreal, 35, a publishing company employee.
"It will take several years for things to calm down. Right now it is so tense you are nervous even talking to your neighbor," Monreal said outside a polling station in Madrid.