SolarWinds hack: the mystery of one of the biggest cyberattacks ever

Back in December, the SolarWinds supply chain attack made the headlines when a Russian cyber espionage group tampered with updates for SolarWinds’ Orion Network Management products that the IT company provides to government agencies, military, and intelligence offices.

A report published by the Washington Post, citing unnamed sources, attributes the attacks to the Russia-linked APT29 cyberespionage group (aka Cozy Bear). The situation immediately appeared critical. After the disclosure of the attack, CISA issued the Emergency Directive 21-01, calling on all federal civilian agencies to review their networks for indicators of compromise power down SolarWinds Orion products immediately.  

The threat actors carried out a highly-sophisticated supply chain attack: SolarWinds networking and security products are currently used by more than 300,000 customers worldwide, including government agencies, military offices, major US telecommunications companies, education institutions, and Fortune 500 companies.

The Pentagon, State Department, NASA, National Security Agency (NSA), Postal Service, NOAA, Department of Justice, and the Office of the President of the United States are clients of SolarWinds.

FireEye was one of the first security firms that investigated the incident. It was also one of the victims of the attack. The security firm confirmed that a threat actor tracked as UNC2452 had used a trojanized SolarWinds Orion business software updates to distribute a backdoor tracked as SUNBURST.

How the SolarWinds supply chain was compromised

Top executives of the SolarWinds firm believe that the root cause of the supply chain attack was an intern who used a weak password for several years. The initial investigation suggested that the password “solarwinds123” was publicly accessible via a misconfigured GitHub repository since June 17, 2018. The issue was addressed on November 22, 2019.

In December, Security researcher Vinoth Kumar revealed he notified the company of a publicly accessible GitHub repository that was leaking the FTP credentials of the company’s download website in the clear text. Threat actors could have used these credentials to upload tainted updates to the SolarWinds download site.

In a hearing before the House Committees on Oversight and Reform and Homeland Security, CEO Sudhakar Ramakrishna confirmed that the password had been in use as early as 2017.

A preliminary investigation revealed that the threat actors behind the SolarWinds attack compromised the SolarWinds Orion supply chain as early as October 2019, but later Crowdstrikes’ researchers dated the initial compromise on September 4, 2019.

“I’ve got a stronger password than ‘solarwinds123’ to stop my kids from watching too much YouTube on their iPad,” Representative Katie Porter of California said. “You and your company were supposed to be preventing the Russians from reading Defense Department emails.”

“I believe that was a password that an intern used on one of his servers back in 2017 which was reported to our security team and it was immediately removed,” Ramakrishna said in response to Porter.

The investigators don’t exclude the use of stolen credentials and brute-force attacks as possible attack vectors.

Confronted by Rep. Rashida Tlaib, former SolarWinds CEO Kevin Thompson declared that the password issue was “a mistake that an intern made.” 

“They violated our password policies and they posted that password on an internal, on their own private Github account,” Thompson explained. “As soon as it was identified and brought to the attention of my security team, they took that down.”

The attack is still ongoing?

According to the experts, the campaign may have begun as early as Spring 2020 and is still ongoing. FireEye discovered multiple weaponized updates that were digitally signed between March and May 2020 and posted to the SolarWinds updates website.

A tainted version of the SolarWinds Orion plug-in masqueraded as the Orion Improvement Program (OIP) protocol. It communicates via HTTP to C2 to retrieve and execute malicious commands, dubbed “Jobs.” The backdoor supports multiple features, including file transferring, executing files, disabling system services, and gathering system info.

The attackers used VPN servers in the same country as the victim to obfuscate the IP addresses and evade detection.

Microsoft also carried out separate analysis and confirmed that the hackers mounted a supply chain attack on SolarWinds. The experts tracked the backdoor as ”Solorigate.”

SolarWinds published a security advisory to disclose the supply chain attack. The company reported the security breach to the authorities and is still investigating the attack with the support of the FBI and security firms. SolarWinds released an update on December 15 to replace the compromised component and implement security enhancements.

According to SolarWinds, up to 18,000 customers may have been impacted by the supply chain attack, including prominent IT and security firms and several Government agencies. The alarming data also emerged in a filing with the Securities and Exchange Commission (SEC).In January, the US authorities officially blamed Russian state-sponsored hackers for the supply chain attack:

“This work indicates that an Advanced Persistent Threat (APT) actor, likely Russian in origin, is responsible for most or all of the recently discovered, ongoing cyber compromises of both government and non-governmental networks.”

reads the statement.

The initial timeline of the SolarWinds attack

In January, Microsoft published a new report that includes additional details of the SolarWinds attack. The analysis shed lights on the handover from the Solorigate DLL backdoor to the Cobalt Strike loader. The attackers focused on separating these two components of the attack chain as much as possible to evade detection. The report provides details regarding the Solorigate second-stage activation that allowed the attacker to deliver Cobalt Strike loaders, such as TEARDROP and Raindrop.

The Solorigate DLL backdoor was compiled at the end of February 2020 and distributed to the potential victims in late March. Then, in June 2020, attackers removed the Solorigate backdoor code from SolarWinds’ build environment.

Experts pointed out that the Solorigate backdoor was designed to stay dormant for at least two weeks, which means that attackers spent about a month selecting the victims and preparing unique Cobalt Strike implants, as well as command-and-control (C2) infrastructure. This means that the “hands-on-keyboard activity” likely started as early as May.

“The removal of the backdoor-generation function and the compromised code from SolarWinds binaries in June could indicate that, by this time, the attackers had reached a sufficient number of interesting targets, and their objective shifted from deployment and activation of the backdoor (Stage 1) to being operational on selected victim networks, continuing the attack with hands-on-keyboard activity using the Cobalt Strike implants (Stage 2),” states the report published by Microsoft.

Microsoft experts analyzed forensic data across the entire environment of impacted organizations to discover how the attackers made lateral movements and how long they remained within their target networks. While investigating the attack, Microsoft identified several second-stage malware and tools, including TEARDROP, Raindrop, and also other custom loaders for the Cobalt Strike beacon.

A third malware strain discovered

Multiple security firms provided more useful information about the attack and discovered additional payloads that were involved in the SolarWinds hack.

In January, researchers from cybersecurity firm CrowdStrike discovered a third malware strain, tracked as SUNSPOT, which was involved in the SolarWinds supply chain attack.

Experts pointed out that even if SUNSPOT was discovered after the Sunburst/Solorigate backdoor and TEARDROP malware, chronologically it may have been the first code to be involved in the attack.

CrowdStrike tracks the threat actor behind the SolarWinds attack as StellarParticle, the group used SUNSPOT to insert the SUNBURST backdoor into software builds of the SolarWinds Orion IT management product.

“SUNSPOT monitors running processes for those involved in compilation of the Orion product and replaces one of the source files to include the SUNBURST backdoor code,” reads the report published by the security firm. “Several safeguards were added to SUNSPOT to avoid the Orion builds from failing, potentially alerting developers to the adversary’s presence.” 

The researchers noticed that once a build command was detected by SUNSPOT, it would insert the malicious code in the Orion app, building a tainted version of the legitimate software. Threat actors spent significant effort in developing the code of SUNSPOT to ensure the stealth injection of the malicious code.

“When SUNSPOT finds the Orion solution file path in a running MsBuild.exe process, it replaces a source code file in the solution directory, with a malicious variant to inject SUNBURST while Orion is being built. While SUNSPOT supports replacing multiple files, the identified copy only replaces InventoryManager.cs,” continues the report.

An updated timeline of the attack

“Our current timeline for this incident begins in September 2019, which is the earliest suspicious activity on our internal systems identified by our forensic teams in the course of their current investigations,” reads the update provided by SolarWinds. “The subsequent October 2019 version of the Orion Platform release appears to have contained modifications designed to test the perpetrators’ ability to insert code into our builds.”

Security experts from Symantec also provided their contribution to the investigation into the attack, revealing the involvement of malware named Raindrop that was used for lateral movement and the deployment of additional payloads.

Raindrop (Backdoor.Raindrop) acts as a loader that was used by attackers to deliver a Cobalt Strike payload. It is similar to the TEARDROP tool, but while the latter was delivered by the initial Sunburst backdoor, the former was used for spreading across the victim’s network.

“Symantec has seen no evidence to date of Raindrop being delivered directly by Sunburst. Instead, it appears elsewhere on networks where at least one computer has already been compromised by Sunburst.”

reads a blog post by Symantec

Symantec investigated four Raindrop infections, and its experts noticed that the malware was employed in the last phases of the attacks against a few selected targets. Both Raindrop and TEARDROP are used to deploy Cobalt Strike Beacon, but they use different packers and different Cobalt Strike configurations. 

In early March 2021, FireEye researchers spotted a new sophisticated second-stage backdoor, dubbed Sunshuttle, that was likely linked to threat actors behind the SolarWinds hack. The malware was discovered while analyzing the servers of an organization that was compromised as a result of the SolarWinds supply-chain attack. The new malware was “uploaded by a U.S.-based entity to a public malware repository in August 2020.”

“Mandiant Threat Intelligence discovered a sample of the SUNSHUTTLE backdoor uploaded to an online multi-Antivirus scan service,” reads the analysis published by Fireeye. “SUNSHUTTLE is a backdoor, written in GO, that reads an embedded or local configuration file, communicates with its C2 server over HTTPS and supports commands including remotely updating its configuration, file upload and download, and arbitrary command execution.”

The SUNSHUTTLE backdoor was likely developed to conduct network reconnaissance alongside other SUNBURST-related tools. Experts pointed out that the SUNSHUTTLE malware was not observed using any trick to gain persistence. This means that the persistence is likely set outside of the execution of this backdoor.

Three new malware tools found

In the same period, Microsoft discovered three new pieces of malware that the threat actors behind the SolarWinds attack, tracked by the IT giant as Nobelium, used as second-stage payloads. The three new malware strains have been tracked as GoldMax, Sibot, and GoldFinder. They were used to maintain persistence and perform malicious actions in very targeted attacks.

The tailor-made malware tools were used as second-stage payloads. The attack vectors were compromised credentials, the SolarWinds binary, lateral movements conducted with the TEARDROP malware, or, in some cases, manually deployed.

The first malware strain, dubbed GoldMax, is a Go-based malware tool used as a command-and-control backdoor by the attackers. The malware used a scheduled task impersonating systems management software as a persistence trick. GoldMax implements a decoy network traffic generator to hide network traffic and avoid detection.

The second malware, dubbed Sibot, is a dual-purpose malicious code written in VBScript used by the threat actors to gain persistence and to download and execute a payload from a remote C2 server.

The third malware is a malware written in Go, dubbed GoldFinder, likely used as a custom HTTP tracer tool that logs the route or hops that a packet takes to reach a hardcoded C2 server.

To be continued?

The investigation is still ongoing and security experts will likely find new pieces of malware involved in the attack.
At the time of writing this report, we cannot exclude that other threat actors might have compromised the SolarWinds supply chain as well or might have had access to the systems initially compromised by SolarWinds hackers.