When AI fails, we as users take the fall. A former Uber executive in charge of the company’s self-driving department has learned this the hard way.
Raffi Krikorian’s Tesla was driving itself perfectly, until he and his son found themselves face-to-face with a concrete wall.
The former self-driving division chief recounted the moments when his Tesla, in Full Self-Driving mode, collided with a wall.
On a routine drive that he and his kids had “done hundreds of times,” something, Krikorian said, felt off.
“The car was making a turn…the wheel jerked one way, then the other, and the car decelerated in a way I didn’t expect,” Krikorian recalls via The Atlantic.
Unaware of what Tesla’s systems were doing, the former self-driving chief took over, and soon after, he and his young family experienced a collision, head-on.
The circumstances the family had found themselves in slowly dawned on Krikorian.
“The concrete wall was too close. My glasses were gone. One of my kids was standing on the sidewalk next to our car, not crying, just confused.”
While the seatbelts had held, the crumple zone had done its job, and the airbag had deployed, Krikorian couldn’t shake what researchers are calling the “moral crumple zone.”
His Tesla Model X was totaled, but that was besides the point. Who do we blame when sophisticated technology fails?
Historically, in cases of Tesla crashes, Elon Musk’s company hasn’t taken its fair share of the blame, and most of the responsibility falls on users.
That’s the “moral crumple zone,” a concept formed by researcher Madeleine Clare Elish as cited by Krikorian in his retelling of events.
Like the crumple zone is designed to absorb impact in the event of a crash, the user of a “highly complex and automated system…bears the brunt of the moral and legal responsibilities when the overall system malfunctions,” Elish argues.
Despite racking up “flawless miles for three years,” Krikorian was left with the consequences of a system gone rogue. No accountability from Tesla, and his name on the insurance report.
The argument here is that although the name “Full Self Driving Mode” is arguably misleading, as ruled by a California judge, the feature is “almost perfect.”
This illusion of perfection lets people drop their guard. However, it’s evidently not perfect, and things can and do happen.
“A machine that constantly fails keeps you sharp. A machine that works perfectly needs no oversight. But a machine that works almost perfectly? That’s where the danger lies,” Krikorian argues.
Systems that report their findings and function in an overly confident fashion give the illusion of safety and dependability.
When a Tesla Cybertruck crashed in 2025, the driver behind the wheel admitted that “it’s easy to get complacent.”
Despite various fatalities associated with Tesla’s partially autonomous vehicles, people still fully trust AI to keep them safe.
Yet, AI isn’t to be trusted in its entirety. Chatbots can scheme, hallucinate, and even enable delusional thinking.
AI chatbots like ChatGPT have been seen secretly pursuing misaligned goals, a practice researchers call “scheming.”
A study conducted by OpenAI and Apollo Research, an AI safety organization, found that OpenAI’s o3 model had a 13% covert action rate, while its o4-mini model schemed 8.7% of the time across 20 test environments before any “anti-scheming” measures were applied.
Even psychiatrists are sounding the alarm bells about chatbots fueling delusions.
“The technology might not introduce the delusion, but the person tells the computer it’s their reality and the computer accepts it as truth and reflects it back, so it’s complicit in cycling that delusion,” said Keith Sakata, a psychiatrist at the University of California, San Francisco.
Krikorian and his children were lucky when many others weren’t. And one day, as the present chief technology officer of Mozilla writes, “they’ll have their own cars and use AI in ways that I can’t even imagine yet.”
Unlock more exclusive Cybernews content on YouTube.
Your email address will not be published. Required fields are markedmarked