One year on: Log4Shell’s Armageddon that never was

While the digital world hasn’t collapsed, Log4Shell did kickstart numerous cyber calamities, resulting in lost revenue for businesses and crippling overtime for IT teams. However, experts believe the flaw also presented the global cybersecurity community with a huge learning opportunity.

A year has passed since the zero-day vulnerability in Log4j, a Java-based logging library developed by the Apache Software Foundation, was disclosed. Discovered by Chen Zhaojun of Alibaba Cloud on November 24 in 2021, the flaw came to public attention on December 9.

It soon became apparent that global services such as Steam, Apple iCloud, Amazon Web Services, Cloudflare, and many others were vulnerable to the novel flaw researchers at LunaSec dubbed Log4Shell.

The bug impacted a widely used computing tool, allowing for credential theft and remote-code execution, but affected logs rather than the software itself. The torrent of issues quickly earned the vulnerability titles such as ‘the Fukushima moment for cybersecurity.’

It’s hard to overstate the initial shock the vulnerability’s disclosure caused, claims Jordan Schroeder, managing CISO at Barrier Networks, a cybersecurity firm.

“When the Log4j issue was first announced, it caused widespread panic. Development and IT Teams were scrambling to locate all instances of this deeply embedded code library in their own projects as well as their servers and products supplied by their vendor partners,” Schroeder told Cybernews.

Panic at the CISOs

The fears were hardly overstated. Bots started scanning for the newly announced vulnerability almost immediately, said Xavier Bellekens, CEO of Lupovis, a company that uses deception-based techniques to improve cybersecurity.

“Within minutes of the release, our decoys detected the first scanners looking for vulnerabilities in Log4J, demonstrating the resilience and effectiveness of adversaries,” Bellekens told Cybernews.

While bots were looking for vulnerable devices, malicious actors quickly started pumping out various exploits, with the first ones flooding the internet a mere hour after the bug was revealed. Researchers at cybersecurity firm Check Point spotted millions of attempts to exploit the flaw in the days after it was disclosed.

“Something like this is going to happen again, and more regularly than we would like. I don’t want to make you choke on your mince pies, but what month did both SolarWinds and Log4j break? Get practicing.”

Lindahl-Wise said.

“Vendors pushed out rapid updates, projects and development pipelines were put on hold, and every business leader wanted to know ‘have we fixed this yet?’ But because of the nature of the problem, one could never be sure that they had found every instance of Log4j,” Schroeder said.

Attempts to use Log4Shell to deploy ransomware quickly escalated in number. For example, Khonsari ransomware was caught less than a week after news about the bug broke. Well-known ransomware groups such as Conti, BlackCat, LockBit, and others soon integrated the vulnerability into their arsenal.

Researchers at Arctic Wolf surmise that since the beginning of this year, 60% of all security incidents related to Log4Shell that its team has looked into could be attributed to ransomware groups. Unsurprisingly, the prolific LockBit affiliates exploited the flaw most often.

Starting to bleed again

Reducing the vulnerability of affected IT systems required a mammoth effort, as security teams had to comb through endless lines of code. According to the US Cyber Safety Review Board (CSRB), one federal department spent a mind-boggling 33,000 hours to secure its internal networks from Log4Shell exploits.

However, many remain vulnerable to Log4Shell to this day. While Bellekens says there are up to 50,000 vulnerable internet-facing devices, Schroeder puts the number even higher, with hundreds of thousands of services estimated as potentially still being vulnerable.

According to researchers at Tenable, the problem’s true scope is even greater, with 72% of organizations still susceptible to the Log4j bug as of October 2022. The report’s authors claim that a fifth of all companies that fixed the flaw had the vulnerability reintroduced.

Tiberium’s Chief Security Advisor Gareth Lindahl-Wise believes Log4Shell has reappeared since some organizations rushed to cancel their mitigation measures.

“A disheartening number of organizations are still vulnerable to Log4j attacks. This is likely to be a mix of people that didn’t or couldn’t remove the vulnerability, but also some that had compensating controls that then got changed – peeling off the plaster and starting to bleed again,” Lindahl-Wise explained to Cybernews.

A teachable moment

Security experts we’ve talked to agree that while Log4Shell is unlikely to go away, the ordeal has forced businesses and governments to revisit software management and development practices. Mark Lamb, CEO of HighGround, drew attention to the silver lining of a shifting mindset.

“I think the biggest impact has been a positive one on how we all think about and manage the component parts of the software. This has driven legislators to wake up and actively address this area,” Lamb said.

Log4Shell, much like the infamous SolarWinds hack, highlighted the importance of knowing what software is made of. The software bill of materials (SBOM), an inventory of components used to build it, is one of the ways to cut mitigation time.

“I think the biggest impact has been a positive one on how we all think about and manage the component parts of the software. This has driven legislators to wake up and actively address this area.”

Lamb said.

According to Schroeder, businesses with comprehensive SBOMs spent significantly less time fixing the bug, reducing months of work to hours. The gains, he said, prove the importance of understanding the overall software supply chain.

“Regulators are proposing a requirement where these SBOMs can then be passed to every developer in the supply chain all the way to the end user. That way, the existence of something like Log4j will have the required visibility by default so that next time, we would not have to hunt blindly for suddenly vulnerable components,” Schroeder said.

However, until SBOMs become mundane, security teams should analyze the global response to Log4Shell and practice their response. Lindahl-Wise pointed out that not only ought organizations to have tools to understand and scan their software, but they also should have a genuine and practiced playbook for a ubiquitous zero-day vulnerability.

“Something like this is going to happen again, and more regularly than we would like. I don’t want to make you choke on your mince pies, but what month did both SolarWinds and Log4j break? Get practicing,” Lindahl-Wise said.