Shaykh Abdallah Adhami said Friday in a joint statement with the center's developer that he will no longer serve as a religious adviser to the center.
Adhami says he wants to devote his time to completing a book on the Quran.
The 44-year-old had been announced as the imam at the center last month after the departure of project co-founder Imam Feisal Abdul Rauf.
His appointment was followed by news reports questioning his views on homosexuality.
Islamic center developer Sharif El-Gamal says he still hoped Adhami would lead prayers at the facility from time to time.