Speaking to a group of Jewish fundraisers, Olmert said he is committed to making the conference a success and said he would meet Palestinian President Mahmoud Abbas on Friday to review preparations. The summit is expected to take place later this year in Annapolis, Maryland.
"If all goes well, hopefully, we will meet in Annapolis," he said. "(But) Annapolis is not made to be the event for the declaration of peace."
Israeli and Palestinian negotiating teams have been trying to draft a joint declaration outlining a future peace agreement ahead of the conference. The teams met Wednesday, but no agreements were announced.
The two sides have expressed vastly different sets of expectations for the summit.
The Palestinians are seeking an agreement addressing the core issues at the heart of the conflict with Israel: final borders, the status of disputed Jerusalem and the fate of Palestinian refugees. They also want a firm timeline for the creation of an independent Palestinian state.
Israel wants a more general document, saying it is premature to address many of these issues.
Olmert has repeatedly warned against expecting too much for the summit. If the gathering falls short of its goals, it could weaken his shaky coalition government and boost Israeli hardliners.
On the Palestinian side, Abbas could suffer in his rivalry with the Hamas militant group. Israel and the international community have been trying to rally behind Abbas since Hamas routed forces from Abbas' Fatah movement and violently seized control of the Gaza Strip in June.
With the sides struggling to bridge the gaps, the U.S. has not yet set a date for the summit or even formally announced who will attend.
Washington hopes that moderate Arab countries that do not have relations with Israel, such as Saudi Arabia, will attend. But Arab countries want guarantees of firm results before committing. Egypt and Jordan, the only Arab countries with peace agreements with Israel, have said the U.S. should consider delaying the summit to give the sides more time to work out their differences.
While Israel has been talking peace with Abbas, tensions in Gaza have been rising.
CBS News correspondent Robert Berger reporting from Jerusalem said Israel's defense minister Ehud Barak has approved limited electricity cuts in Gaza in response to Palestinian rocket attacks. Under the plan, Israel will cut electricity for increasing lengths of time each time a rocket is fired across the Gaza border. The "punishment" would begin with short cutoffs of 15 minutes, and increase in length if more rockets fall.
Israel hopes the sanctions will build pressure on militants to halt their near-daily rocket attacks on southern Israel. It remains unclear when the sanctions would go into effect.
Despite the Israeli threats, Palestinians fired two rockets into southern Israel Thursday morning, the military said. No damage or casualties were reported.
Israel provides more than half of Gaza's electricity, so any cutoff would add to the misery in the impoverished coastal area. However, the cuts would not result in a total blackout.
Members of the international community, including U.N. chief Ban Ki-moon, have condemned the Israeli plan as unfair collective punishment. But with its military unable to stop the crude projectiles, Israel says it is running out of options to deal with the rocket fire. Barak has said a large ground offensive into Gaza - something Israel has tried to avoid - will likely be necessary at some point.
Israel's government last month declared Gaza a "hostile territory," clearing the way for Israel to impose the sanctions.
On Wednesday the Israeli military killed two members of a Palestinian rocket squad, Hamas security and hospital officials said. The militant Islamic Jihad said it fired rockets at Israel a few minutes before the Israeli attack. It said the two dead were 15 years old. Often teenagers go out to get launchers after rockets are fired.
The Israeli military said it hit militants after rocket fire from the area.
Early Thursday, Israeli soldiers killed two Hamas militants in a clash near the Gaza city of Khan Younis, Hamas officials and the military said. The army said the dead men were carrying an anti-tank missile launcher and assault rifles.
Also Thursday, the Israeli army said it opened fire at a group of militants planting explosives near the Gaza border with Israel. Islamic Jihad said two members were killed. Israel said it arrested one militant.
It was unclear what effect an Israeli cutoff of power would have on Gazans, who already suffer frequent electricity shortages. Many buildings have backup generators.
In the West Bank, Israeli activists blocked a major road west of Jerusalem, delaying traffic and scuffling with police, demanding the highway be opened for Palestinian use.
The road, linking Jerusalem with central Israel, dips into the West Bank for about 10 kilometers (six miles) along its route, but Palestinians are barred from using any part of it for fear they could launch attacks on Israelis. Police spokesman Mickey Rosenfeld said three protesters were arrested.