Troops held a ceremony that included lowering the Australian flag, which had flown over Camp Terendak in Talil, southern Iraq, a Department of Defense spokesperson said, speaking on condition of anonymity as required by the department.
Australia, a staunch US ally, was one of the first countries to commit troops to the Iraq war five years ago.
But Rudd was swept to office last November on the promise to bring Australia's 550 combat troops home by the middle of this year.
Several hundred other troops will remain in Iraq to act as security and headquarters liaisons and to guard diplomats.
Australia will also leave behind two maritime surveillance aircraft and a warship to help patrol oil platforms in the Gulf.
Rudd said the Iraq deployment has made Australia more of a target for terrorism.
The soldiers and 65 army trainers were stationed at Talil, about 185 miles south of Baghdad, and were responsible for providing security training for Iraqi forces, as well as reconstruction and aid work. They were also on standby to offer backup to Iraqi forces in the south for the past two years.
In February, the head of Australia's defense force, Air Chief Marshal Angus Houston, told a Senate inquiry that the troops were no longer needed in Iraq.
The combat troops were expected to return home over the next few weeks. Australian media reports said the first of the soldiers had already landed in the country Sunday afternoon.
Rudd remains committed to keeping Australia's 1,000 troops in Afghanistan.