"We discussed ways forward, and one of the issues we discussed was the formation of a security subcommittee that would address at a expert or technical level some issues relating to security, be that support for violent militias, al Qaeda or border security," Ambassador Ryan Crocker said after the meeting that included lunch and spanned nearly seven hours.
Iraqi Foreign Minister Hoshyar Zebari said experts would meet as early as Wednesday to work out the structure and mechanism of the committee.
"We hope that the next round of talks will be on a higher level if progress is made," he said at a separate news conference after the talks.
But underscoring the rising tensions between the two arch-foes, Crocker reiterated accusations that Iran is fueling the violence in Iraq by arming and training Shiite militias. He warned no progress could be made unless Iranian actions change on the ground.
"The fact is, as we made very clear in today's talks, that over the roughly two months since our last meeting we've actually seen militia-related activity that could be attributed to Iranian support go up and not down," Crocker said, citing testimony from detainees and weapons and ammunition confiscated in Iraq as evidence.
"We made it clear to the Iranians that we know what they're doing (and) it's up to them to decide what they want to do about it," he said.
For his part, Iranian Ambassador Hassan Kazemi Qomi countered that Tehran was helping Iraq deal with the security situation but Iraqis were "victimized by terror and the presence of foreign forces" on their territory.
He said his delegation also demanded the release of five Iranians detained by U.S. forces in Iraq. The United States has said the five were linked to Iran's elite Quds Force, which it has accused of arming and training Iraqi militants. Iran says the five are diplomats who were legally in Iraq.
"There are also Iranian citizens who have been detained on legally entering Iraq. We demanded their release too. We discussed the creation of a mechanism to implement what we achieved in the first round of talks. They (the Americans) acknowledged making mistakes and this is a step forward in itself and it's now up to the Americans to rectify their mistakes," Qomi said.
"We are hoping that you support stability in Iraq, an Iraq that doesn't interfere in the affairs of others nor want anyone to meddle in its own affairs," he said, according to excerpts of al-Maliki's remarks released by his office.
"It's Iraq's right to call on everyone to stand beside it to counter the scourge of terror and extremism," he said. "The world ... must stand together and face this dangerous phenomenon and its evils, which have gone beyond the borders of Iraq after terror and al Qaeda groups received strong blows and are now running away from the fight and moving to other nations."
An Iraqi official who was present at the meeting room said Crocker and Qomi were involved in a heated exchange early in the talks.
It began when Crocker confronted the Iranians with charges that Tehran was supporting Shiite militiamen killing U.S. troops, providing them with weapons and training. Qomi dismissed the allegations, saying the Americans had no proof, according to the official, who spoke on condition of anonymity because he wasn't authorized to disclose the information.
"It is not surprising that the talks got off to a rocky start, because few U.S. diplomats involved in the talks expect Tehran to stem the flow of money and arms into Iraq," said CBS News Foreign Affairs Analyst Pamela Falk.
"But the point of the talks is broader, and that is to open channels of communication to see if, at some point, the Iranians might see it in their interest to find a diplomatic solution to the sectarian violence as well as agree to a compromise on their nuclear program — one of the other important issues that is not on the agenda at this week's talks."
In other developments: