This comes the same day that retired Marine Corps Gen. John Kelly was.
The New York Times first reported the development, which said that Kelly asked Mr. Trump to remove Scaramucci from his role. The report added that Kelly also made clear Monday morning that he's now in charge.
Scaramucci had previously bragged that he would report directly to the president rather than the chief of staff.
At a Cabinet meeting earlier in the day, Mr. Trump said, "I predict that General Kelly will go down, in terms of the position of chief of staff, one of the great ever. And we're going to have a good time, but much more importantly, we're going to work hard and we're going to make America great again."
Mr. Trump named Kelly, who had been serving as Homeland Security secretary, to the new role on Friday, replacing Reince Priebus. Priebus said that he had resigned from the position, which he had served in since the inauguration in January.
Scaramucci was viewed as a provocative figure who bumped heads with both Priebus and Sean Spicer,that Scaramucci was chosen to lead the White House communications operation. Sarah Huckabee Sanders has since taken over as press secretary.
Last week, he came under fire for. He said that he believed that Priebus had leaked information about the dinner he attended with the president that night.
"They'll all be fired by me," he said. "I fired one guy the other day. I have three to four people I'll fire tomorrow. I'll get to the person who leaked that to you. Reince Priebus -- if you want to leak something -- he'll be asked to resign very shortly...Reince is a f******* paranoid schizophrenic, a paranoiac."
Then using a vulgar colloquialism, Scaramucci said that Priebus had tried to block him for months, and was now giving Fox executive Bill Shine the same treatment. The entire exchange between Lizza and Scaramucci was outlined in a story published online Thursday by the New Yorker.
Scaramucci is the second communications director to leave the position. In May, Michael Dubke resigned from the gig, saying that he was leaving for personal reasons.