The ruling overturned a lower judge's September decision, and sided with the league in a battle with the players union.
With the suspension in place, Brady will not be playing in games against the Arizona Cardinals, Miami Dolphins, Houston Texans, and Buffalo Bills.
The three-judge panel of the 2nd U.S. Circuit Court of Appeals in Manhattan ruled 2-to-1 that Commissioner Roger Goodell did not deprive Brady of "fundamental fairness" with his procedural rulings. The split decision may end the legal debate over the scandal that led to months of football fans arguing over air pressure and the reputation of one of the league's top teams.
It is also likely to fuel a fresh round of debate over what role, if any, the quarterback and top NFL star played in using underinflated footballs at the AFC championship game in January 2015. The Patriots won the contest over the Indianapolis Colts, 45-7, and then won the Super Bowl.
The ruling can be appealed to the full 2nd Circuit or to the U.S. Supreme Court, but it would likely be a steep and time-consuming climb even if the courts took the unusual step to consider it.
In a majority opinion written by Judge Barrington D. Parker, the 2nd Circuit said its review of labor arbitration awards "is narrowly circumscribed and highly deferential — indeed, among the most deferential in the law."
"Our role is not to determine for ourselves whether Brady participated in a scheme to deflate footballs or whether the suspension imposed by the Commissioner should have been for three games or five games or none at all. Nor is it our role to second-guess the arbitrator's procedural rulings," the opinion said. "Our obligation is limited to determining whether the arbitration proceedings and award met the minimum legal standards established by the Labor Management Relations Act."
The 2nd Circuit said the contract between players and the NFL gave the commissioner authority that was "especially broad."
"Even if an arbitrator makes mistakes of fact or law, we may not disturb an award so long as he acted within the bounds of his bargained-for authority," the court said.
Meanwhile, the issue addressed in the ruling went beyond deflated footballs. The league had argued that Brady, a four-time Super Bowl winner, tarnished the game by impeding the NFL's investigation.
Specifically, the panel noted that Brady destroyed a cellphone containing nearly 10,000 messages, as CBS2's Otis Livingston reported.
Judge Parker said the cellphone destruction raised the stakes "from air in a football to compromising the integrity of a proceeding that the commissioner had convened.... So why couldn't the commissioner suspend Mr. Brady for that conduct alone? With all due respect, Mr. Brady's explanation of that made no sense whatsoever."
In a dissent, Chief Judge Robert Katzmann said Goodell failed to even consider a "highly relevant" alternative penalty.
"I am troubled by the Commissioner's decision to uphold the unprecedented four-game suspension," Katzmann said. "It is ironic that a process designed to ensure fairness to all players has been used unfairly against one player."
The NFL Players Association said in a statement it was disappointed.
"We fought Roger Goodell's suspension of Tom Brady because we know he did not serve as a fair arbitrator and that players' rights were violated under our collective bargaining agreement," the statement said. "Our union will carefully review the decision, consider all of our options and continue to fight for players' rights and for the integrity of the game."
NFL spokesman Brian McCarthy said the court ruled Goodell acted properly in cases involving the integrity of the game.
"That authority has been recognized by many courts and has been expressly incorporated into every collective bargaining agreement between the NFL and NFLPA for the past 40 years," McCarthy said.
In the September decision that the ruling overturned, Manhattan U.S. District Judge Richard Berman went against the league. He let Brady skip the suspension last season.
Goodell insisted the suspension was deserved.
The appeals court settled the issue well before the start of the 2016 season, avoiding the tension built last year when Brady didn't learn until a week before the season that he would be allowed to start in the Patriots' opener.
At oral arguments in March, appeals judges seemed skeptical of arguments on Brady's behalf by the NFL Players Association.
Circuit Judge Denny Chin said evidence of ball tampering was "compelling, if not overwhelming" and there was evidence that Brady "knew about it, consented to it, encouraged it."
The league argued that it was fair for Goodell to severely penalize Brady after he concluded the prize quarterback tarnished the game by impeding the NFL's investigation by destroying a cellphone containing nearly 10,000 messages.
Parker said the cellphone destruction raised the stakes "from air in a football to compromising the integrity of a proceeding that the commissioner had convened."
"So why couldn't the commissioner suspend Mr. Brady for that conduct alone?" he asked. Parker added: "With all due respect, Mr. Brady's explanation of that made no sense whatsoever."
Parker also was critical of the NFL at the arguments, saying Brady's lengthy suspension seemed at "first blush a draconian penalty."
Former teammate Willie McGinest said on the NFL Network that he doesn't believe Brady will continue to fight the NFL.
"I'm listening to people talk about fighting tooth-and-nail. Tom is exhausted. This was an emotional roller coaster for him, his teammates, his family the last year. And all he wants to do is just move on and play football. He thought he was over it," McGinest said on the NFL Network Monday.
McGinest continued, "But I think he's over it. He's going to take his time, he's going to do whatever the league wants him to do and just move on from the whole situation."
As 1010 WINS' Al Jones reported, CBS Sports Radio's Jay Berman said the NFL needed the win.
"That proves to everybody else that there isn't special treatment, even though there really is," Berman said.
Some New Yorkers on Monday were less than sympathetic toward Brady in the wake of the ruling.
"I think it's fair. He should serve the suspension," said Eddie Rivera of Brooklyn. "You know, he's not above all the other players."
"I think it's well-deserved," said Marshall Baker of Brooklyn. "I mean, it's not the first time the Patriots have gotten away with, you know, some… minor discrepancy, and you know, he got to play last season without no suspension, and now he's got to pay for it."
"Kind of funny the Patriots are finally getting caught for something," said Sebastian Cray of East Harlem. "I mean, they seem to get caught for stuff all the time, but they always seem to sort of get away with it or not get punished as harshly. So, I don't know, it seemed like a pretty small crime deflating footballs, but destroying the evidence seemed kind of bad. So maybe it's good they finally got him for something."
In an interesting twist, the Patriots and Brady agreed to a two-year contract extension in the offseason. In essence, CBS2's Livingston explained, Brady's four-game suspension will only cost $200,000 instead of $2 million.
(TM and © Copyright 2016 CBS Radio Inc. and its relevant subsidiaries. CBS RADIO and EYE Logo TM and Copyright 2016 CBS Broadcasting Inc. Used under license. All Rights Reserved. This material may not be published, broadcast, rewritten, or redistributed. The Associated Press contributed to this report.)
for more features.