Court papers filed in federal court in Washington by a State Department investigator show two Algerian women came forward separately to say they had been sexually assaulted by the man while at his home in Algiers, where he was stationed.
The court documents identify the man as Andrew Warren. He has not been charged with a crime. An affidavit submitted by an investigator in November sought to search Warren's computer for evidence of past sexual assaults.
A congressional official familiar with the case said Wednesday the man was employed by the CIA in Algiers until he was recalled in the wake of the investigation.
The official spoke on condition of anonymity because he was not authorized to discuss the investigation. The case was first reported by ABC News.
The affidavit is careful not to say exactly who the man works for, other than the U.S. government. The document says Warren has been assigned to Algiers since 2007.
One of the Algerian women claimed that she was drinking at a party at the man's home when something made her suddenly ill and she passed out. She awoke believing she had had intercourse, but with no memory of having done so.
The affidavit says a second woman approached embassy staff in 2008 claiming that the man had sexually assaulted her at his home. Both women reported the incidents months after they occurred.
State Department spokesman Robert Wood said the man in question has returned to the United States while the government investigates.
"The United States takes very seriously any accusations of misconduct involving any U.S. personnel abroad," said Wood, referring further questions to the Justice Department, which declined to comment.