The source would not provide further details Wednesday.
When contacted by CNN, Brafman said he had no comment.
Weinstein is accused of raping a woman in a New York hotel room in 2013 and forcibly performing oral sex on another woman at his Manhattan apartment in 2006.
He faces five felony charges: two counts of predatory sexual assault, one count of criminal sexual act in the first degree and one count each of first-degree rape and third-degree rape. Weinstein has pleaded not guilty.
The case hasn't been completely smooth for either side. In October, one of Weinstein's six felony charges was dismissed after an NYPD officer was found to have mishandled evidence.
In a series of interviews for a profile by Esquire, Brafman spoke about representing Weinstein.
"I've spent forty years trying to get to the top of my profession, and this is the most high-profile case in the United States," Brafman said. "It's flattering to be picked by someone like him."
Brafman also addressed how the #MeToo movement has impacted Weinstein's high-profile case and others.
"One day it's Charlie Rose. Then it's Kevin Spacey. Matt Lauer. Les Moonves. Suddenly they have something about Judge Kavanaugh from when he was in high school? Give me a break," Brafman told the magazine. "I have a daughter; I have granddaughters. I'd like them to not have to deal with harassment, discrimination, and an unequal pay scale. But a movement becomes dangerous when it generates the kind of hatred that keeps a person from getting a fair trial."
When speaking to CNN in August, Brafman said it would be "difficult but not impossible" to seat an impartial jury in Weinstein's trial.
It's not yet known who would take over for Brafman.
A representative for Weinstein declined to comment.
for more features.