Last Updated May 20, 2015 4:12 PM EDT
BEIRUT -- Militants with the Islamic State of Iraq and Syria seized almost full control of the ancient town of Palmyra in central Syria on Wednesday after fierce clashes with government troops, renewing fears the extremist group would destroy the priceless archaeological site if it reaches the ruins.
The Britain-based Syrian Observatory for Human Rights said the militants gained control of almost the entire town in heavy clashes during the day. A media collective for Palmyra also said that ISIS was now in control of most of the town.
Palmyra is home to a UNESCO World Heritage Site and is famous for its 2,000-year-old majestic Roman colonnades. It wasn't immediately clear how close the militants were to the famed ruins, which are just southwest of the town.
CBS News' George Baghdadi reports from Damascus that the government said it had evacuated hundreds of citizens to safe territory. Officials also said hundreds of statues had been moved to safety, Baghdadi reports.
State-run Syrian TV said in a terse statement that an "evacuation by Syria's National Defence Forces followed heavy battles in and around the central city," Baghdadi reports.
The majority of the ruins are located in Palmyra's south, and the militants entered Wednesday from the north. There were conflicting reports over whether the militants had seized the state security building, or prison, from government forces. Several online sources of information about the extremist group reported the militants had freed 45 prisoners.
Their presence has sparked concerns they would destroy the ruins as they have done with major archaeological sites in neighboring Iraq.
Following setbacks in both Syria and Iraq, ISIS fighters appear to have gotten a second wind in recent days, capturing Ramadi, the capital of Iraq's largest Sunni province, and advancing in central Syria to the outskirts of Palmyra.
In Iraq, thousands of displaced people fleeing from Ramadi and the violence in the western Anbar province poured into Baghdad on Wednesday after the central government waived restrictions and granted them conditional entry, a provincial official said.
The exodus is the latest in the aftermath of the fall of Ramadi - Anbar's provincial capital - to ISIS over the weekend. The Shiite-led government in Baghdad is struggling to come up with a plan to reverse the stunning loss of the city, pledging a counter-offensive and relying on Iranian-backed Shiite militiamen to join the battle.
Athal al-Fahdawi, an Anbar councilman, said that thousands of civilians from Ramadi who were stranded on open land for days are now being allowed to cross a bridge spanning the Euphrates River and enter Baghdad province.
On Tuesday, Anbar local officials said five of the displaced residents had died from exhaustion in Bzebiz area, where the displaced had been forced to stay while they were kept away from Baghdad.
According to the International Organization for Migration, more than 40,000 people have been displaced from Anbar since Friday, when ISIS conquered Ramadi. In the past, people fleeing the Sunni province have been prevented from entering Baghdad due to fears that militants might mingle in with the crowds and sneak into the Iraqi capital.
Residents still left in Ramadi told The Associated Press over the phone Wednesday that ISIS militants were urging them over loudspeakers not to be afraid and to stay in the city. However, they were not preventing those wanting to leave the city to go, the residents said, speaking on condition of anonymity for fear for their own safety.
It is still unknown when the expected wide-scale operation to recapture Ramadi and other cities will start. Baghdad officials and leaders of the so-called Popular Mobilization Units, which consists of a number of Shiite militias who are fighting on the side of the Iraqi military and security forces, have repeatedly said they need time for a military buildup and reconnaissance.
When ISIS launched its blitz last year and entire cities and towns fell into the hands of the militants, the Iraqi government at first took only defensive measures and in many cases, soldiers and Iraqi forces abandoned their posts and fled in the face of the ISIS assault.
Military operations to retake entire swaths of Iraq that had fallen to ISIS began only months later. The U.S. launched its airstrikes campaign in August.
Meanwhile, police officials said a roadside bomb hit a minibus in Baghdad's northern Shiite suburbs of Husseiniyah, killing two passengers and wounding seven. Also, a bomb near a wholesale market in southern Baghdad killed two people and wounded five, the officials said, speaking on condition of anonymity because they were not authorized to talk to the media.
Nobody immediately claimed responsibility for the attacks, but Islamic militants frequently claim such bombings. Elsewhere in Baghdad, three bodies with gunshot wounds to their head and chest were found Wednesday in different parts of the city. The police said there were no IDs with the bodies.
Dead bodies left in the street were a common occurrence during the widespread sectarian violence that engulfed Iraq several years ago.