Experts who wrestled with SolarWinds hackers say cleanup could take months

Cybersecurity expert Steven Adair and his team were in the final stages of purging the hackers from a think tank's network earlier this year when a suspicious pattern in the log data caught their eye.

The spies had not only managed to break back in – a common enough occurrence in the world of cyber incident response – but they had sailed straight through to the client's email system, waltzing past the recently refreshed password protections like they didn't exist.

"Wow," Adair recalled thinking in a recent interview. "These guys are smarter than the average bear."

It was only last week that Adair's company - the Reston, Virginia-based Volexity - realized that the bears it had been wrestling with were the same set of advanced hackers who compromised Texas-based software company SolarWinds.

Using a subverted version of the company's software as a makeshift skeleton key, the hackers crept into a swathe of U.S. government networks, including the Departments of Treasury, Homeland Security, Commerce, Energy, State and other agencies besides.

When news of the hack broke, Adair immediately thought back to the think tank, where his team had traced one of the break-in efforts to a SolarWinds server but never found the evidence they needed to nail the precise entry point or alert the company. Digital indicators published by cybersecurity company FireEye on Dec. 13 confirmed that the think tank and SolarWinds had been hit by the same actor.

Senior U.S. officials and lawmakers have alleged that Russia is to blame for the hacking spree, a charge the Kremlin denies. 

Adair – who spent about five years helping defend NASA from hacking threats before eventually founding Volexity – said he had mixed feelings about the episode. On the one hand, he was pleased that his team's assumption about a SolarWinds connection was right. On the other, they had been at the outer edge of a much bigger story. 

A big chunk of the U.S. cybersecurity industry is now in the same place Volexity was earlier this year, trying to discover where the hackers have been and eliminate the various secret access points the hackers likely planted on their victims' networks. Adair's colleague Sean Koessel said the company was fielding about 10 calls a day from companies worried that they might have been targeted or concerned that the spies were in their networks.

His advice to everyone else hunting for the hackers: "Don't leave any stone unturned."

Koessel said the effort to uproot the hackers from the think tank - which he declined to identify - stretched from late 2019 to mid-2020 and occasioned two renewed break-ins. Performing the same task across the U.S. government is likely to be many times more difficult.

"I could easily see it taking half a year or more to figure out - if not into the years for some of these organizations," Koessel said. 

Pano Yannakogeorgos, a New York University associate professor who served as the founding dean of the Air Force Cyber College, also predicted an extended timeline and said some networks would have to be ripped out and replaced wholesale. 

In any case, he predicted a big price tag as caffeinated experts were brought in to pore over digital logs for traces of compromise.

"There's a lot of time, treasury, talent and Mountain Dew that's involved," he said.

(Reporting by Raphael Satter; Editing by Andrea Ricci)