Washington — U.S. government agencies were ordered to scour their networks for malware and disconnect potentially compromised servers after authorities learned that thein a months-long global cyberespionage campaign. The campaign was discovered when a prominent cybersecurity firm learned it had been breached.
In a rare emergency directive issued late Sunday, the Department of Homeland Security's cybersecurity arm warned of an "unacceptable risk" to the executive branch from a feared large-scale penetration of U.S. government agencies that could date back to mid-year or earlier.
"This can turn into one of the most impactful espionage campaigns on record," said cybersecurity expert Dmitri Alperovitch.
The hacked cybersecurity company, FireEye, wouldn't say who it suspected - many experts believe the operation is Russian, given the careful tradecraft - and noted that foreign governments and major corporations were also compromised.
News of the hacks, first reported by Reuters, came less than a week after FireEye disclosed that nation-state hackers had broken into its network and stolen the company's own hacking tools.
On Sunday, Russia's U.S. embassy described as "unfounded" in a post on its Facebook page the "attempts of the U.S. media to blame Russia for hacker attacks on U.S. governmental bodies."
The apparent conduit for the Treasury and Commerce Department hacks - and the FireEye compromise - is a hugely popular piece of server software called SolarWinds. It's used by hundreds of thousands of organizations globally, including most Fortune 500 companies and multiple U.S. federal agencies that will now be scrambling to patch up their networks, said Alperovitch, the former chief technical officer of the cybersecurity firm CrowdStrike.
On its website, SolarWinds says it has 300,000 customers worldwide, including all five branches of the U.S. military, the Pentagon, the State Department, NASA, the National Security Agency, the Department of Justice and the White House. It says the 10 leading U.S. telecommunications companies and top five U.S. accounting firms are also among customers.
The DHS directive - only the fifth since such directives were created in 2015 - said U.S. agencies should immediately disconnect or power down any machines running the impacted SolarWinds software.
The government's Cybersecurity and Infrastructure Security Agency (CISA) said it was working with other agencies to help "identify and mitigate any potential compromises." The FBI said it was engaged in a response but declined to comment further.
FireEye, without naming any specific targets, said in a blog post that its investigation into the hack of its own network had identified "a global campaign" targeting governments and the private sector that, beginning in the spring, had slipped malware into a SolarWinds software update. Neither the company nor the U.S. government publicly identified Russian state-backed hackers as responsible.
The malware gave the hackers remote access to victims' networks, and Alperovitch said SolarWinds grants "God-mode" access to a network, making everything visible.
"We anticipate this will be a very large event when all the information comes to light," said John Hultquist, director of threat analysis at FireEye. "The actor is operating stealthily, but we are certainly still finding targets that they manage to operate in."
FireEye said it had confirmed infections in North America, Europe, Asia and the Middle East, including in the health care and oil and gas industry - and had been informing affected customers around the world in the past few days. Its customers include federal, state and local governments and top global corporations.
It said any infiltration of an infected organization required "meticulous planning and manual interaction."
That means it's a good bet only a subset of infected organizations were being spied on by the hackers. Nation-states have their cyberespionage priorities, which includevaccine development.
In a tweet Sunday, former CISA chief Chris Krebs said "hacks of this type take exceptional tradecraft and time," adding he believed its impact was only beginning to be understood.
In a later tweet, he applauded CISA's order:
He also tweeted that, "While (CISA) directives only apply to Feds, everyone else should follow suit."
Krebs was fired from his CISA post by President Trump last month after Krebs vouched for the integrity of the presidential election and disputed Mr. Trump's claims of widespread election fraud.
Microsoft on Sunday issued "guidance" to help cybersecurity professionals fight "increased activities from a sophisticated threat actor that is focused on high value targets such as government agencies and cybersecurity companies. We believe this is nation-state activity at significant scale, aimed at both the government and private sector." Microsoft said none of its products had been hit.
The Treasury Department referred requests for comment to the National Security Council, whose spokesman, John Ullyot, said the government was "taking all necessary steps to identify and remedy any possible issues related to this situation."
The intrusions disclosed Sunday included the Commerce Department's agency responsible for internet and telecommunications policy. A spokesperson confirmed a "breach in one of our bureaus" and said "we have asked CISA and the FBI to investigate."
Austin, Texas-based SolarWinds confirmed Sunday a "potential vulnerability" related to updates released between March and June for software products called Orion that help monitor networks for problems.
"We believe that this vulnerability is the result of a highly-sophisticated, targeted and manual supply chain attack by a nation state," said SolarWinds CEO Kevin Thompson said in a statement. He said it was working with the FBI, FireEye and intelligence community.