The prisoners escaped last week, apparently by digging a tunnel, some 180 yards long, that emerged in the women's section of a mosque near the headquarters, the security officials said.
It was not the first major prison escape for al Qaeda militants. At least four members of the group broke out of a prison at Bagram, the main U.S. base in Afghanistan in July. Among them was Omar al-Farouq, a top leader of al Qaeda in Southeast Asia.
The capture, trial and imprisonment of those responsible for the attack on the USS Cole was a high point in Yemen's declared alliance with the United States in the war on terror, reports CBS News correspondent Mark Phillips. The escape of those behind the attack is a deep embarrassment and puts Yemen's determination into question.
An investigation headed by the Yemeni interior minister has begun questioning intelligence officers on how Friday's escape in San'a happened, government and security officials said, speaking on condition of anonymity because they were not authorized to divulge details on the probe.
"It couldn't have happened without the coordination of high ranking officers in the intelligence," said one official. He pointed to possible infiltration of the intelligence agency by militants, saying hundreds of Yemenis who fought in Afghanistan in the 1980s against Russian occupation were given jobs with the security forces when they returned home.
"It is no surprise that many of these former fighters are sympathetic to al Qaeda," he said.
Yemen's interior ministry confirmed in a statement that the convicts escaped from the headquarters for the political security forces, a security branch affiliated with the intelligence. But it gave no further details on the escape or the search for the militants.
The prison was underground in the headquarters, one of the most heavily guarded buildings in the capital. The compound is surrounded by a high wall and armed guards and all roads leading to it are blocked to vehicles.
The 23 militants, all convicted members of Al Qaeda, were all kept in the same cell, the officials said.
The international police agency Interpol said Sunday it was informed by Yemen that Jamal al-Badawi — a man convicted of plotting, preparing and helping carry out the Cole bombing — was among the fugitives. Al-Badawi was among those sentenced to death in September 2004 for plotting the attack, in which two 17 sailors were killed when suicide bombers blew up an explosives-laden boat next to the destroyer as it refueled in the Yemeni port of Aden on Oct. 12, 2000.
His sentence was later reduced to 15 years in prison.
For at least 10 of the escaped convicts, this is the second time they escaped prison. In 2003, they escaped briefly from a prison in the city of Aden before they were recaptured and convicted.
The first time al-Badawi escaped, he headed for Yemen's remote tribal lands, and officials assume that is where he is headed this time, reports CBS News correspondent David Martin. The FBI has offered to help in the manhunt, but so far the Yemenis have not taken up the offer, Martin reports.
Checkpoints throughout the city have been set up and the interior ministry has issued photographs of the convicts to security forces throughout the country to aid in their capture.
Since Friday, at least a dozen members of Islamic groups in Yemen known to be affiliated with the convicts, have been detained for interrogation. Some relatives of the convicts have also been summoned for questioning.
Authorities have also approached tribal leaders to watch for the fugitives, particularly in Maarib, a lawless region northeast of the capital.
In Afghanistan, a search for the four al Qaeda members who escaped in July is still continuing, said U.S. military spokesman Lt. Mike Cody said. Military officials declined to say how they broke out of the high-security facility at Bagram.
The four boasted about their breakout on a video believed filmed in Afghanistan and broadcast in October on Dubai-based TV station Al-Arabiya. They claimed they picked a lock and timed the escape for a Sunday when many of the Americans on the base were off duty.