"If Pakistan were to deteriorate further, the control of those weapons would be up for grabs," said Stephen Cohen of the Brookings Institute. "And we don't want to see that happen."
That could trigger an unthinkable cataclysm - if those weapons were taken by a rogue state or a terror group. And that's the second worry. Al Qaeda is a re-invigorated enemy that's poised to take advantage of the turmoil.
"It has to increase the threat against the U.S. because it gives al Qaeda precisely the breathing space, the opportunistic moment in time now that they will have now because of the distraction because of the instability in Pakistan," said terrorism expert Bruce Hoffman of Georgetown University.
Six years after 9/11, it's Pakistan, not Afghanistan or Iraq, that's the central front in the president's war on terror.
Top U.S. intelligence analysts say al Qaeda has re-established a central command inside "a safe haven" in the tribal regions of Pakistan - with top leaders continuing to train fresh recruits and plot new attacks.
Al Qaeda operatives, trained in Pakistan, have already launched numerous terror plots, including the suicide bombings of the London subway and the failed plans to blow up US bound airliners and strike U.S. targets in Germany.
In the early months after 9/11, Pakistan proved to be a strong ally, helping to capture top terrorists, including Sept. 11 attack plotter Ramzi Bin al-Shibh.
But, recent efforts by Pakistani troops to take out al Qaeda have failed. And with the Musharraf government now worried about its own survival, the fight against bin Laden's forces is not its top priority.
"This might be an environment that al Qaeda might try to do something staged out of Pakistan, but directed toward the West," said Former Director of the National Counterterrorism Center John Brennan, who is a CBS News Analyst.
U.S. security officials say they're not hearing any increased "chatter" about new threats. But as the instability in Pakistan grows, so do their worries.