As CBS News Correspondent Sharyl Attkisson reports, it's not just what he saw that disturbed him, it's what he didn't see: the top secret X Division, arguably the most important and powerful division at the lab.
If the X Division sounds familiar, it might be from past scandals. It's where the government claims China obtained secrets to the United States' most sophisticated nuclear warhead, the W-88. It's also where two computer hard drives with sensitive nuclear weapons data disappeared.
And the X Division is the source of the physics behind U.S. nuclear weapons, where some of the best scientists in the world do their most important work.
Brown was stunned to find no evidence the X Division had ever been audited properly. When his team got the assignment, they didn't even get a foot in the door when an X Division manager blocked them.
Brown recalls being told, "You can't come in, we don't recognize your authority."
He says it's "absolutely not" proper for the group being audited to hold authority over the auditor.
In the end, lab managers backed the X Division.
Auditors were only allowed to review paperwork offsite. Still, they concluded the system might not "capture items that may pose a potential safety problem" and lapses in training and qualifications put "the effectiveness of the nuclear weapons program ... at risk."
Brown laid out his concerns in reports sent to lab and Department of Energy officials.
A spokesman for the National Nuclear Security Administration who asked to be unnamed insisted the nation's nuclear stockpile, which includes the W76 and W88 warheads, remains "safe and reliable."
"The Secretaries of Energy and Defense provide the President an annual assessment of the safety, security, and reliability of the nuclear weapons stockpile,'' the spokesman said. "Any potential issues are thoroughly investigated."
But one government official tells CBS News that hundreds of problems, including some in the X Division, were addressed during the lab's lengthy shutdown over separate security concerns.
Today, U.S nukes are no longer tested in full-scale explosions, making it all the more critical, Brown argues, that the X Division's work gets proper scrutiny.
Without it, Brown says, "You can't say with confidence that those weapons will perform their designed function."
Brown says that makes the X Division audit that didn't happen an issue of national security.