CBS News National Security Correspondent David Martin reports the arrests began over the weekend after a tip from U.S. intelligence.
The men were allegedly preparing to carry out what one official called a "well-planned" sneak attack on the American embassy.
A senior U.S. official, who asked not to be named, said it was not clear how an attack would be carried out. But the source said the basis for concern was suspicious movement in the vicinity of the embassy by people who appeared to be gathering information about security and other aspects of the embassy operation.
The activity around the embassy prompted a June 9 warning to U.S. visitors in Yemen to take precautions and forced the closing of the embassy to the public.
When the threat persisted, the FBI decided to withdraw its personnel Sunday, concerned that violence might be directed at them. FBI officials were in Yemen investigating the attack on the USS Cole.
The plot comes in the wake of guilty verdicts against four men for the bombings of two American embassies in Africa.
Prosecutors said those bombings were directed by suspected terrorist mastermind Osama bin Laden and officials believe but cannot prove he is connected to this latest plot as well.
It was not clear if there was any connection between the suspected Yemen plot and arrests in India of three men who were allegedly planning to bomb U.S. embassies in India and Bangladesh.
One State Department official said, "This is one we were certainly worried about." But now that the arrests have been made, officials say, the immediate threat of an attack is over.
CBS News State Department Reporter Charles Wolfson reports State Department spokesman Richard Boucher said the FBI made the decision to pull out on June 17 "based on what they saw as a credible threat to their employees."
The USS Cole was refueling in the Yemeni port of Aden when a small harbor skiff pulled alongside it and detonated explosives that killed 17 sailors, injured 39 others and nearly sank the ship.
The FBI investigators in the USS Cole case are from the bureau's New York field office. More than 30 arrests have been made in the case so far.
The embassy will be closed through Wednesday of this week and officials will re-evaluate the security situation then.
While some family members have departed Yemen, so far no government employees have left.
"Although our embassy is closed to the public our embassy does remain open and our diplomats continue to do their jobs," Boucher said Monday. "The department has sent additional diplomatic security personnel to help the embassy and to ensure that all personnel receive appropriate protection."
Those security agents arrived last week. Boucher also said that the Yemeni government had taken "extraordinary steps" because of the heightened threat.
Yemeni authorities have provided additional personnel to augment perimeter security at the embass, the ambassador's residence, other diplomatic residential areas and other places frequented by Americans.
In Jordan, Yemen's Foreign Minister Abu Baker Al-Qorabi told the Middle East Broadcasting Corp. Tuesday his government had given Americans all necessary protection.
An office set up in Aden to investigate the bombing was shut recently and its U.S. investigators moved to the U.S. embassy in San'a. An embassy official has said the closure was not related to new security concerns.
The embassy plot is apparently in response to only the latest security scare in the Gulf.
In May, the United States issued a global warning against terrorism to Americans travelling abroad, and the Pentagon upgraded security at its naval headquarters in Bahrain and for some of its forces in Kuwait, because of a "reliable and specific threat."
Those moves coincided with the embassy bombing convictions, which mean possible death sentences for two alleged associates of bin Laden.
In October, in the wake of the attack on the Cole, U.S. forces in Bahrain and Qatar were placed on the highest state of alert in response to what Pentagon officials called specific and credible intelligence reports of plans for a terrorist attack.
© MMI Viacom Internet Services Inc. All Rights Reserved. This material may not be published, broadcast, rewritten, or redistributed. The Associated Press contributed to this report