The suspects, picked up Friday and Saturday as part of an FBI sting operation, are due in separate courtrooms Tuesday for their arraignment on charges of conspiring to provide material support to al Qaeda.
Tarik Shah, a martial arts expert in New York, is accused of offering to train so-called holy warriors - including showing them how to use prayer beads as deadly weapons.
Rafiq Abdus Sabir, a Florida physician, was going to ship out to work at a Saudi military base this week. But before he could hop that plane, federal agents picked him up and accused him of planning to use his military base credentials to move about freely to act as the doctor for any wounded members of al Qaeda.
If convicted, each man faces a maximum sentence of 15 years in prison and a fine of up to $250 thousand.
New York police spokesman Paul Browne said Shah was arrested early Friday. Florida authorities said Sabir was arrested Saturday.
News of Sabir's arrest spread quickly through the gated community where he lives with his family and where neighbors in the previous few days had noticed strangers in the neighborhood and wondered what was afoot.
"We wondered what they were doing," one neighbor, Ian McDonald, told CBS News. "Then the neighbors found out it was the FBI, they were watching somebody in the neighborhood but we had no real idea who it was."
Sabir, 50, is being held at the Palm Beach County Jail; prosecutors have not revealed where Shah, 42, is being held.
It was not immediately known whether either man had an attorney.
Prosecutors said Sabir agreed to treat jihadists, or holy warriors, in Saudi Arabia, while Shah agreed to train them in hand-to-hand combat and at one point demonstrated how his "prayer beads could be used to strangle a person."
Authorities also claim that Shah told the undercover agent that since he was young, training jihad fighters had always been one of his dreams.
The one-count complaint details a sting operation beginning in 2003 in which the two men took an oath pledging their allegiance to al Qaeda.
Prosecutors say both suspects met more than once with a confidential source and an undercover agent posing as an al Qaeda operative and recruiter. Authorities say many of those meetings were recorded.
Prosecutors said Shah presented himself and Sabir as a "package deal."
Shah allegedly indicated he wanted to train fighters "to wage jihad" and to "find people who were willing to press the fight," prosecutors said.
Authorities say Shah inspected a warehouse in Long Island to determine if it was suitable as a training facility, and he described his previous attempts to recruit others, which included a trip to Phoenix. He said he would give a training manual for hand-to-hand combat and a videotape to the undercover agent so that others in the Middle East could see his "usefulness to the cause," prosecutors said.
Shah also allegedly said he and Sabir tried to attend training camps in Afghanistan in 1998. He also is known as Tarik Ibn Osman Shah, Tarik Jenkins and Abu Musab, prosecutors said.