The four occupants -- one a member of South Korea's National Olympic Committee -- were freed unharmed a short time later, and the escaped convicts abandoned the car and fled on foot.
The car was seized at Silverwater, a Sydney suburb adjacent to Olympic Park and the site of the large minimum-security Silverwater prison, shortly before 4 p.m., according to police.
None of the occupants was taken hostage in the escape, said Sydney Olympics organizing committee spokesman Milton Cockburn. All four were taken to a hospital, apparently to be treated for shock.
One of the occupants was a low-ranking official of the South Korean Olympic committee and the other three were Korean volunteers from Australia, said Chou Eun-ki, a spokesman for the delegation.
Australian Associated Press quoted police sources as saying one of the occupants of the van was a pregnant woman. There were unconfirmed reports that the pregnant women was taken to a specialist maternity hospital in Sydney later Tuesday for treatment.
A spokesperson for the New South Wales corrective services department said one of the prisoners was a 26-year-old man who had been on weekend detention for stolen property offenses. He had breached his weekend conditions and was in prison full time.
The spokesman said a 35-year-old man also involved in the incident still had 18 months to go on an undetermined sentence for armed robbery and culpable driving causing death.
Chou said South Korean officials were still getting details of the hijacking from the four, but the hijacking apparently was foiled by police.
The car was later dumped in the inner-city suburb of Marrickville. Police launched a manhunt and were using dogs to track the escaped convicts.
Last March, an inmate was plucked from the same prison by a hijacked helicopter and escaped as guards opened fire in what the head of the corrective services department described as "a very daring, almost Hollywood-type escape."