Shopping baskets blocking self-serve checkouts, crowds at airports, banks offline – widespread IT outages caused by a faulty update have sent multiple systems haywire.

Multiple reports of major IT system outages are coming from major Australian institutions and other parts of the world. The outage is affecting Windows machines, and users on social media post images of computers crashing and showing a “blue screen of death,” or BSOD.

CrowdStrike, a global cybersecurity vendor, confirmed that outages are caused by a defective update and said it is actively working with impacted customers. The defect was found “in a single content update for Windows hosts.” Mac and Linux hosts are not impacted.

“This is not a security incident or cyberattack. The issue has been identified and isolated, and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website,” George Kurtz, CrowdStrike's President and CEO, posted on X.

The company also recommends that organizations ensure they’re communicating with CrowdStrike representatives through official channels.

“Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.”

CrowdStrike's share price is down 12% today in pre-market trading.

The Australian Cybersecurity Coordinator previously explained on X that the outages relate “to a technical issue with a third-party software platform employed by affected companies.”

I am aware of a large-scale technical outage affecting a number of companies and services across Australia this afternoon.

Our current information is this outage relates to a technical issue with a third-party software platform employed by affected companies.

undefined National Cyber Security Coordinator (@AUCyberSecCoord) July 19, 2024

Downdetector reports reveal multiple outages affecting not only Microsoft services, but also banks, broadband networks, news agencies, cloud and other services.

Australian public broadcaster ABC linked the incident to CrowdStrike, as the issues are affecting organizations that have CrowdStrike Falcon installed in their environments.

TECH OUTAGE: Self service machines across Woolworths supermarkets are not operational. Blue screen of death. #crowdstrike pic.twitter.com/RS42zcEQi2

undefined Archie Staines (@archiestaines9) July 19, 2024

This preliminary assessment was also provided by CyberCX, an Australian cybersecurity firm, and a spokesperson for Australia's home affairs minister.

CrowdStrike's deployed software update is causing affected systems to experience a recurring boot cycle, ultimately resulting in BSOD. CrowdStrike Falcon is a cloud-based threat checker used by many businesses.

Airlines grounded, train schedules disrupted

Downdetector has revealed that major American airlines are also experiencing issues. Spikes of outages are being reported at Delta Air Lines, American Airlines, United Airlines, and others.

The US Federal Aviation Administration has stopped Delta, United and American Airlines flights due to a “communication issue.”

Berlin Brandenburg Airport shared a notice to passengers that due to technical problems there may be delays.

Passagierhinweis: Aufgrund einer technischen Störung kommt es zu Verzögerungen bei der Abfertigung.

Information for passengers: Due to a technical fault, there will be delays in check-in. pic.twitter.com/gVint8DqiS

undefined BER – Berlin Brandenburg Airport (@berlinairport) July 19, 2024

The outage has also hit the Hong Kong airport, forcing airlines to perform check-in manually, South China Morning Post reports. The Airport Authority said it has activated its emergency response mechanism.

Ryanair said that due to disruptions across the network due to “a Global 3rd party IT outage, which is entirely out of our control,” booking and check-in are currently unavailable.

“If you are due to travel today (19th July) and have yet to check-in for your flight, you can do so at the airport.”

BBC reports that a major train company in the UK has warned passengers to expect disruptions due to “widespread IT issues.” Govia Thameslink Railway's brands Southern, Thameslink, Gatwick Express, and Great Northern posted on social media, “We are currently experiencing widespread IT issues across our entire network.”

"We are unable to access driver diagrams at certain locations, leading to potential short-notice cancellations, particularly on the Thameslink and Great Northern networks,” the post reads.

US emergency service computers have also been affected, Forbes reports. One commenter on Hacker News said their entire emergency department was knocked offline.

Delta Airlines in ATL hit pic.twitter.com/SrghLWFcbF

undefined Andrew Chiles (@AndrewChiles) July 19, 2024

“Really scary when you have ambulances coming in and are trying to stabilize a heart attack. 911 is down in Oregon too,” the comment reads.

Broadcasters from Australia and the UK are also affected as ABC and Sky News said they were experiencing diffuculties.

The Microsoft / CrowdStrike outage has taken down most airports in India. I got my first hand-written boarding pass today 😅 pic.twitter.com/xsdnq1Pgjr

undefined Akshay Kothari (@akothari) July 19, 2024

Microsoft said it investigating issues and taking “mitigating actions,” according to AFP news agency.

Meanwhile, workarounds to solve the issue are being shared on social media, as CrowdStrike posted a Tech Alert for its logged-in consumers only.

Crowdstrike fix. May the force be with you. Always. pic.twitter.com/h63aoqUN1u

undefined `Mike D (we/us) (@mike_d_ok) July 19, 2024

“Workaround Steps: Boot Windows into Safe Mode or the Windows Recovery Environment; Navigate to the C:\Windows\System32\drivers\CrowdStrike directory; Locate the file matching “C-00000291*.sys”, and delete it; Boot the host normally,” the tech alert reads.

We're aware of an issue with Windows 365 Cloud PCs caused by a recent update to CrowdStrike Falcon Sensor software. This is being communicated under WP821561 in the admin center. (Cont...)

undefined Microsoft 365 Status (@MSFT365Status) July 19, 2024

One of the biggest outages ever

Troy Hunt, a security consultant who runs data-breach search website Have I Been Pwned, already called the incident “the largest IT outage in history.”

“This is basically what we were all worried about with Y2K, except it's actually happened this time,” he said.

The same was posted by hacker group Anonymous: “The current outage is one of the biggest ever.”

“CrowdStrike declaring an early weekend by taking out half the world’s systems. Even ransomware isn’t this effective,” one security engineer tweeted on X.

How does the update cause a system reboot cycle?

Elliott Wilkes, CTO of Advanced Cyber Defence Systems, explains that the disruption in service on Windows devices, which is affecting customers in industries across the globe, from airlines in India and Australia to Sky News in the UK, appears to be caused by an error introduced in an update file pushed yesterday by cybersecurity company CrowdStrike in their Falcon product.

“This tool has software that runs on end-user devices – called an “agent” – and runs in a similar fashion to classic antivirus software running on a desktop computer,” Wilkes told Cybernews.

Because agent-based detection systems often require enhanced or even administrator-level privileges to monitor computer activity and detect malicious code, they can introduce risks.

Tools like Falcon typically have the ability to take action to immediately resolve or suspend services if malicious activity is detected. This is a hugely important feature and needed component in order to rapidly defend against attacks, prevent infiltrators from moving laterally across an organization, Wilkes explains.

“However, these enhanced permissions come with risks because they are integrated into critical components of the operating system of the end-user devices. What we are seeing here is end-user devices getting stuck in a reboot loop on a screen known as the “blue screen of death,” the infamous Windows error screen. Ultimately, the likelihood of these events is small, but the impact, as we can see today, is tremendous,” Wilkes said.

“Given the specific nature of this failure, getting stuck without the ability to reboot, makes this situation particularly challenging to resolve."

Adrianus Warmenhoven, cybersecurity advisor at NordVPN, believes CrowdStrike’s update did not undergo A/B testing, which suggests that it was not supposed to change any major parts of the code. However, small changes might have broken the code.

“Given the amount of systems that are down... it is hard to not look at the Quality Assurance team or the testing team. Drivers are usually near the privileged parts in an OS (because of performance reasons) and security drivers are all the way up there, close to the maximum privilege level. They have to be otherwise malware might circumvent them (this, of course, has happened before),” Warmenhoven said.

“It was deployed all over the world at around the same time. So that tells me, they were either confident that nothing major changed, or the QA team messed up.”

Many affected organizations, despite their ISO certifications, failed in their disaster recovery plans once their machines became unbootable. Some may have never tested their backup strategies.

“We see a lot of assumptions being tested today, and a lot of them being proven false.

And since we are converging to fewer and fewer vendors, just like with biology, a single issue can cause bigger problems,” Warmenhoven concluded.

One vulnerability should not have global ramifications

Wil Jones, Technical Director at Propel Tec, said that CrowdStrike is software used to prevent data breaches, however, this incident is a stark reminder of the dangers of our increased reliance on cloud-based software. Many vital systems, relying on a single cloud-based software, fell like dominoes.

“The size of this global IT outage is quite something. So far we know it is impacting everything from GP appointment booking systems to Gail's bakery ordering systems, and of course, grounding flights in New Zealand and America. Right now depending on where you are in the world an IT glitch means that you can't buy a croissant, see a doctor or catch a flight,” said Jones. “Here in the UK, the most worrying issue appears to be the EMIS outage, which has got GPs handwriting prescriptions.”

He believes this incident should give central governments globally a pause for thought.

“When one vulnerability can have such global ramifications, should a situation like this ever be exploited by a rogue nation-state or terrorist organisation, it could have devastating consequences globally,” Jones warns. “Whether this is an intentional outage or just an unfortunate glitch, it needs to be seen as a canary in the coalmine moment.”

While CrowdStrike seems to be the root cause of the problem, Rupert Brown, the CTO at Evidology Systems, believes that Microsoft will also have to add many new test cases to its QA processes in the future to mitigate the BSOD failures. Businesses should also review their recovery procedures, as most of them now are plugin catchups.

“The problem we actually face today is that infrastructure is heavily virtualized and has dynamic topology, so it is harder to test for particular instances,” Brown said. “Few if any global businesses know at any point in time during the day what machine instances and network topology/protocols are active, architecture teams have been lulled into a false sense of security about these issues.”

He advocates for better change management, testing, and physical and virtual topology tracking.

“Regulators do need to act but to be candid they cannot afford to have staff of sufficient caliber working for them. In the same way that we have many accounting scandals due to bad financial auditing it is likely that the security practices within many large consulting companies are just as flawed and need major scrutiny,” Brown said.

Updated on July 19th [07:30 a.m. GMT] with additional information.

Updated on July 19th [08:30 a.m. GMT] with additional information.

Updated on July 19th [11:05 a.m. GMT] with a statement from Crowdstrike and additional information.

Updated on July 19th [11:40 a.m. GMT] with additional comments.