Gerhard Schroeder refused to concede defeat and said he could still theoretically remain in power if talks with other parties were successful.
Merkel claimed her party had received a mandate from voters to form a new coalition government, and she would talk to all parties with the exception of a small left-wing group as she tried to become Germany's first female chancellor.
"What is important now is to form a stable government for the people in Germany, and we...quite clearly have the mandate to do that," she said.
Left Party leader Oskar Lafontaine ruled out forming an alliance with Schroeder's party, shutting out the chance of an all-left coalition.
Sunday's vote centered on different visions of Germany's role in the world and how to fix its sputtering economy. Schroeder touted the country's role as a European leader and counterbalance to America, while Merkel pledged to reform the moribund economy and repair ties with Washington.
An exit poll by ZDF public television showed Merkel's Christian Democrats at 35.9 percent, considerably worse than expected and short of a majority with her preferred coalition partners, the pro-business Free Democrats, at 10.4 percent.
Schroeder's Social Democrats were at 33.6 percent, according to ZDF.
ARD public television showed almost identical results, with Merkel's party at 35.7 percent and the Social Democrats at 33.7 percent.
Merkel's party consistently polled above 40 percent during the campaign.
Without a majority, Merkel may be forced as chancellor into a coalition with the Social Democrats, probably without Schroeder.