AT&T outage due to failed network update, FCC releases new findings

The US Federal Communications Commission (FCC) has finished its investigation into February’s hours-long AT&T outage that disrupted dozens of 911 call centers nationwide and says the telecom giant’s lack of “best practices” and numerous “procedural errors” led to the public safety fiasco.

The February 22nd outage of 5G voice and data services – which lasted approximately 12 hours – blocked more than 92 million phone calls and more than 25,000 attempts to reach 911 emergency services, the FCC said in a new report issued on Monday.

In an ominous foreshadowing of last week’s CrowdStrike outage, the AT&T snafu was revealed to be the result of a network update gone wrong, the FCC detailed in the 10-page report, citing “numerous factors” having which contributed to the “extensive scope and duration of the outage.”

“When you sign-up for wireless service, you expect it will be available when you need it – especially for emergencies,” said FCC Chairwoman Jessica Rosenworcel.

“This ‘sunny day’ outage prevented consumers across the country from communicating, including by blocking 911 calls, and stopped public safety personnel from using FirstNet,” Rosenworcel said, vowing to “provide accountability and prevent similar outages in the future.”

We are aware of the reported wireless outages, and our Public Safety and Homeland Security Bureau is actively investigating. We are in touch with AT&T and public safety authorities, including FirstNet, as well as other providers.

undefined The FCC (@FCC) February 22, 2024

Emergency centers impacted nationwide

FirstNet or the First Responder Network Authority is a specialized communication network built in partnership with AT&T, which launched the priority LTE spectrum band in 2017, locking in a 25-year contract to manage it.

The FirstNet system serves about 27,500 public safety agencies and organizations and is the first network to ever use tower-to-core encryption based on open industry standards, AT&T’s website shows.

"As a result of this outage, not only were members of the public without a reliable connection to 911 services, but they were unable to reach family members, employers, health care professionals, schools, and other forms of assistance."

- Federal Communications Commission on AT&T outage

Besides disrupting calls and text messaging for 290 million customers covered by AT&T's 5G wireless network, the outage caused major problems in dozens of cities whose 911 emergency services went down.

And, although a cyberattack was quickly ruled out, the company struggled to find the reason behind the failure at the time, leaving government officials, and the public, in the dark for hours.

"As a result of this outage, not only were members of the public without a reliable connection to 911 services, but they were unable to reach family members, employers, health care professionals, schools, and other forms of assistance, " the report said.

Numerous inadequacies found

The FCC report listed eight mitigating factors either causing the massive event or impacting the recovery process:

• configuration errors,
• lack of adherence to AT&T Mobility’s internal procedures,
• lack of peer review,
• failure to adequately test after installation,
• inadequate laboratory testing,
• insufficient safeguards and controls to ensure approval of changes affecting the core network,
• lack of controls to mitigate the effects of the outage once it began,
• multiple system issues that prolonged the outage once the configuration error had been remedied.

The FCC stressed the need for adherence to internal procedures and industry best practices when implementing network changes.

It chastised AT&T for failing to already have network controls in place to fix potential configuration errors “so they do not escalate and disrupt network operations.”

The report also blamed the telecommunications company for failing to have “appropriate systems and procedures in place with adequate capacity to facilitate prompt recovery from large-scale outages.”

The FCC is recommending further action, referring AT&T to the US Public Safety and Homeland Security’s Enforcement Bureau for potential violations.

In 2021, T-Mobile was forced to pay $19.5 million to regulators after a 12-hour outage led to more than 20,000 failed 911 emergency calls during the height of the COVID-19 pandemic.

The agency has also launched a separate investigation into the telecom for its conduct during an April ransomware attack that exposed the personal data of ‘nearly all’ 109 million AT&T customers.