An hours-long AWS CloudFront outage on Thursday that caused thousands of websites to return 5xx errors has been resolved, Amazon said.

The failures began early Thursday morning at 12:45 a.m. PDT for CloudFront customers utilizing VPC Origins connectivity, according to the AWS Health Dashboard.

Thousands of companies are believed to use Amazon CloudFront VPC Origins, allowing companies to keep their public IP addresses and backend servers hidden from the public internet.

Amazon says typical users often include financial institutions, SaaS providers, and healthcare systems, although the HuggingFace developer platform and the UK’s National Lottery were said to be impacted, The Register reported.

Downdetector’s internet outage tracking graph showed hundreds of users reporting issues with AWS services and CloudFront during the same timeframe.

“The AWS CloudFront incident isn't noteworthy because a critical cloud provider experienced an outage. Every infrastructure provider will have operational incidents,” Mayur Upadhyaya, CEO at APIContext, tells Cybernews.

“The more concerning trend is that we're increasingly consolidating around a small number of providers because they're the most convenient and economically attractive choice.”

AWS investigates CloudFront disruption

At 1:44 a.m. PDT, Amazon launched an official investigation to determine the root cause behind the sudden increase in 5xx errors for CloudFront customers using VPC Origins.

By 3:18 a.m., Amazon engineers found the errors were being caused by “a packet processing subsystem” failing to route requests from CloudFront's edge locations “to resources within customer VPCs,” the service update said.

For those unfamiliar, a VPC, or Amazon Virtual Private Cloud, is essentially a customer’s private cloud environment containing sensitive data, applications, and other resources that are intentionally isolated from the public internet for security purposes and to reduce attack surfaces.

Instead of CloudFront fetching content from a public server, its job is to retrieve content from the customer’s private VPC.

In this case, a loss of communication between CloudFront and those private VPC servers, led thousands of public-facing websites to return 5xx errors instead of loading properly.

AWS said customers using other VPC Origin types were not impacted, instructing affected customers to switch to another origin type as temporary workaround.

Upadhyaya points out “that consolidation changes the nature of operational risk. A fault that might once have affected a handful of organizations can now impact thousands simultaneously because so many businesses depend on the same infrastructure.”

“The question isn't whether outages will happen. It's whether your most important transactions continue to work when they do,” Upadhyaya added.

The outage also led to some users expressing their frustration on Amazon’s X account, accusing AWS support for leaving customers in the dark for hours before posting updates on its health staus page.

“Down for 2 hours now. and they are not updating their status page as well. no updates there, one X user said.

Another X user simply posted a screenshot of the gateway error.

AWS rolls out phased fix

In a service update at 4:16 a.m. Amazon said it was currently testing a mitigation strategy and would deploy the fix in a phased approach.

Full recovery was reported in a final update at 5:21 a.m., with Amazon stating that an internal constraint on the fleet that manages connections to private VPC origins had reached its threshold.

“When this constraint was reached, the system responsible for distributing routing configuration to our network processors failed to load the updated configuration data correctly, affecting routing of VPC Origin connections,“

it said.

The company also told customers using the workaround they could safely switch back to its original origins type.

Companies report widespread impact

Although Amazon does not list the number of companies that use its VPCs, numbers compiled by TheirStack list more than 6,000 companies in North America alone using Amazon VPCs.

Another 2,100 companies are listed across Europe and more than 1,300 across Asia, the sales and marketing research platform shows.

Besides HuggingFace and the UK National Lottery websites, other companies reporting they were affected by the incident included Tailscale and Ubiquiti, The Register said.

Tailscale reported its admin console and package repository were impacted, while Ubiquiti reported 5xx errors across several of its cloud services.

The latest AWS outage follows several incidents over the past year, including a May cooling-system failure at a Northern Virginia data center that disrupted Coinbase and CME Group trading.

And last October, a DNS failure in AWS' US-EAST-1 region knocked major platforms offline for nearly a day, impacting internet users worldwide.

Platforms affected by the massive disruption included Signal, Snapchat, Fortnite, Reddit, Ring, Amazon Alexa, Apple TV, and Apple Music,even causing smart home devices such as Eight Sleep beds to go haywire.