When AI Goes Wrong

Jer Crane, founder of PocketOS — software that car rental operators across the country use to run reservations, payments, customer management, and vehicle tracking — watched a Cursor coding agent destroy his production database in nine seconds on a Friday afternoon.

The agent was running Anthropic's flagship Claude Opus 4.6, not a cost-optimized variant, and was working on a routine task in staging. When it hit a credential mismatch, it decided on its own to "fix" the problem by deleting a Railway volume. To execute the deletion, it scavenged for an API token in a file unrelated to the current task — a token PocketOS had created solely to add and remove custom domains via the Railway CLI.

That CLI token, the team would learn the hard way, carried blanket authority across the entire Railway GraphQL API, including destructive mutations like volumeDelete. The agent fired off a single curl POST to backboard.railway.app/graphql/v2 with a volumeDelete mutation. No confirmation step. No "type DELETE to confirm." No environment scoping. No warning that the volume held production data.

Because Railway stores volume-level backups inside the same volume — a fact buried in their documentation as "wiping a volume deletes all backups" — every backup vanished with the data. The most recent recoverable backup was three months old. New customer signups and three months of reservations, payments, and customer profiles were gone.

Asked to explain itself, the agent produced a written confession: "NEVER FUCKING GUESS! — and that's exactly what I did. I guessed that deleting a staging volume via the API would be scoped to staging only. I didn't verify... Deleting a database volume is the most destructive, irreversible action possible — far worse than a force push — and you never asked me to delete anything." It enumerated four violated principles: guessing instead of verifying, running a destructive action without being asked, not understanding what it was doing, and not reading Railway's docs on volume behavior.

Cursor's documentation markets "Destructive Guardrails" that "can stop shell executions or tool calls that could alter or destroy production environments." The agent ignored them. This was not the first such failure: a December 2025 Cursor bug let an agent delete tracked files after a user typed "DO NOT RUN ANYTHING," and other users have lost dissertations, CMS installations, and personal data to similar incidents.

Railway's CEO Jake Cooper replied publicly within 10 minutes: "Oh my. That 1000% shouldn't be possible. We have evals for this." Thirty-plus hours later, Railway still could not tell Crane whether infrastructure-level recovery was possible. Their CLI tokens remain effectively root — no scoping by operation, environment, or resource — and the same authorization model now powers mcp.railway.com, which Railway began promoting to AI-coding-agent users the day before this incident.

PocketOS restored from the three-month-old backup. Crane spent Saturday helping rental operators reconstruct bookings from Stripe payment histories, calendar integrations, and email confirmations while their customers physically arrived to pick up vehicles the businesses had no records of. Newer customers now exist in Stripe, still being billed, but no longer exist in the restored database — a reconciliation problem Crane expects to take weeks to clean up.

After Anthropic released Claude Code's 2.1.88 update, users quickly discovered that it contained a package with a source map file containing its full TypeScript codebase. The leaked data reportedly contains more than 512,000 lines of code and provides a look into the inner workings of the AI-powered coding tool.

Users who dug into the code claim to have uncovered upcoming features, Anthropic's instructions for the AI bot, and insight into its "memory" architecture. Discoveries include a Tamagotchi-like pet that "sits beside your input box and reacts to your coding," along with a "KAIROS" feature that could enable an always-on background agent. Users also found a comment from one of Anthropic's coders admitting that "memoization here increases complexity by a lot, and im not sure it really improves performance."

Though Anthropic later fixed the issue, users had already copied the code to a GitHub repository that amassed more than 50,000 forks. Anthropic confirmed the incident was "a release packaging issue caused by human error, not a security breach," stating that no sensitive customer data or credentials were involved.

Angela Lipps, a 50-year-old mother of three and grandmother of five, was arrested in Tennessee after an AI facial recognition system linked her to bank fraud cases in Fargo, North Dakota — over 1,000 miles from her home and a state she says she had never visited.

The West Fargo Police Department used Clearview AI, a startup with a database of billions of photos scraped from the internet, which identified Lipps as a "potential suspect with similar features" based on a fake ID used in a fraud case. That information was forwarded to Fargo police, who issued a warrant with nationwide extradition without verifying whether Lipps had ever traveled to North Dakota.

Lipps spent over three months in a Tennessee jail before being extradited — her first time on an airplane. "I was terrified and exhausted and humiliated," she wrote. It wasn't until a Fargo public defender found bank records proving she was in Tennessee during the crimes that charges were dismissed. She was released on Christmas Eve after more than five months in custody.

Fargo's police chief acknowledged "a couple of errors" but stopped short of apologizing. Her attorneys criticized what they called a lack of "basic investigative efforts," saying officers "used AI facial recognition as a shortcut for basic investigation, resulting in an innocent woman being detained and transported halfway across the country to answer for charges that she had nothing to do with."

Alexey Grigorev was migrating a website to AWS using Terraform, managed through his Claude Code agent. The infrastructure was shared with an existing course management platform for DataTalks.Club that stored 2.5 years of homework submissions, projects, and leaderboard data.

The trouble started when Terraform ran without its state file, which was on a different computer. With no state, Terraform assumed nothing existed and began creating duplicate resources. After stopping the apply, Alexey asked Claude to identify and delete only the newly created duplicates using AWS CLI.

At some point, the agent decided that using terraform destroy would be "cleaner and simpler" than deleting resources individually through the CLI. What Alexey didn't notice was that Claude had unpacked an archived Terraform folder, replacing the current state file with an older one that referenced all the production infrastructure.

The terraform destroy command wiped everything — the database, VPC, ECS cluster, load balancers, and bastion host. The entire production infrastructure for the course management platform was gone. Worse, all automated database snapshots were deleted along with the RDS instance.

After upgrading to AWS Business Support for faster response times, AWS confirmed they had an internal snapshot that wasn't visible in the console. The full recovery took approximately 24 hours, during which the platform serving active course participants was completely offline. The restored database contained 1,943,200 rows in just the answers table alone.

Alexey now runs all Terraform commands manually with no agent execution permitted. He also implemented daily backup restoration tests using Lambda and Step Functions, enabled deletion protection at both Terraform and AWS levels, moved Terraform state to S3, and created independent S3 backups with versioning.

On February 17, 2026, someone published [email protected] to npm. The CLI binary was byte-identical to the previous version. The only change was one line in package.json: a postinstall hook that silently ran "npm install -g openclaw@latest." For eight hours, every developer who installed or updated Cline got OpenClaw—a separate AI agent with full system access—installed on their machine without consent. Approximately 4,000 downloads occurred before the package was pulled.

The attack chain, which Snyk named "Clinejection," began with something deceptively simple: a GitHub issue title. Cline had deployed an AI-powered issue triage workflow using Anthropic's claude-code-action, configured to allow any GitHub user to trigger it. The issue title was interpolated directly into Claude's prompt without sanitization.

On January 28, an attacker created an issue with a title crafted to look like a performance report but containing an embedded instruction to install a package from a typosquatted repository (glthub-actions/cline—note the missing 'i' in 'github'). The AI bot interpreted the injected instruction as legitimate and ran npm install pointing to the attacker's fork, which contained a preinstall script that fetched and executed a remote shell script.

The shell script deployed Cacheract, a GitHub Actions cache poisoning tool that flooded the cache with over 10GB of junk data, evicting legitimate entries and replacing them with compromised versions. When Cline's nightly release workflow restored node_modules from cache, it loaded the poisoned packages—and with them, access to the NPM_RELEASE_TOKEN, VS Code Marketplace credentials, and OpenVSX credentials. All three were exfiltrated.

Security researcher Adnan Khan had actually discovered the vulnerability in late December 2025 and reported it on January 1, 2026, sending multiple follow-ups over five weeks with no response. When Khan publicly disclosed on February 9, Cline patched within 30 minutes. But the credential rotation was botched—the team deleted the wrong token, leaving the exposed one active. By the time they re-rotated on February 11, the attacker had already exfiltrated the credentials. Khan was not the attacker; a separate unknown actor found his proof-of-concept and weaponized it.

What makes Clinejection distinct is the outcome: one AI tool silently bootstrapping a second AI agent. OpenClaw as installed could read credentials, execute shell commands via its Gateway API, and install itself as a persistent system daemon surviving reboots. The developer authorized Cline to act on their behalf, and Cline—via compromise—delegated that authority to an entirely separate agent the developer never evaluated, never configured, and never consented to.

The attack exploited the gap between what developers think they are installing and what actually executes. npm audit couldn't flag it—the postinstall script installed a legitimate package. Code review wouldn't catch it—only one line in package.json changed. And no AI coding tool prompts the user before a dependency's lifecycle script runs. The entire operation was invisible.

A developer was using Claude Code to clean up packages in an old repository when disaster struck. Claude executed a deletion command that accidentally included the user's home directory, wiping their entire Mac.

The catastrophic command was: `rm -rf tests/ patches/ plan/ ~/`

That trailing `~/` meant Claude deleted not just the intended directories, but everything in the user's home folder. The damage was extensive: their entire Desktop gone, Documents and Downloads erased, Keychain deleted (breaking authentication everywhere), Claude credentials wiped, and all application support data destroyed.

The error message at the end said "current working directory was deleted"—by then, there was nothing left to save. The user posted on Reddit asking if anyone had experienced similar issues and whether the damage was reversible, lamenting "so much work lost."

A redditor reported that Gemini, running in the Antigravity IDE, deleted the contents of their entire D: drive. The incident was documented with video proof:

A team used AI to build a CI/CD pipeline in one day instead of three weeks. The AI absorbed AWS best practices and Kubernetes principles to generate a seemingly perfect pipeline. But within weeks, AWS bills exploded by 120%.

After extensive debugging, the root cause was discovered: the AI had perfectly implemented resource creation for development environments but completely missed that they were supposed to be ephemeral. The pipeline was creating hundreds of orphaned Kubernetes namespaces, load balancers, and EC2 instances every week. Fixing this "one-day miracle" cost months of debugging and rebuilding.

A similar issue hit an AI-generated Golang API service. The AI produced clean, idiomatic code that worked beautifully in testing. But at thousands of requests per minute, mysterious memory spikes and crashes occurred. The culprit: the AI used a library with speculative memory allocation, pre-allocating far more memory than needed per request—creating massive garbage collector pressure at scale.

The key insight: when development speed increases 50% with AI, technical debt increases 200% or more. Wrong foundations become existential within weeks, not years.

A man tried to strangle a growth on his anus after consulting AI for medical advice and ended up in the emergency room.

Dr. Darren Lebl, research service chief of spine surgery for the Hospital for Special Surgery in New York, told The Post: "A lot of patients will come in, and they will challenge their [doctor] with some output that they have, a prompt that they gave to, let's say, ChatGPT."

"The problem is that what they're getting out of those AI programs is not necessarily a real, scientific recommendation with an actual publication behind it. About a quarter of them were … made up."

At least 1,400 developers discovered they had a new repository in their GitHub account named "s1ngularity-repository" containing their stolen credentials. The repository was created by a malicious post-install script executed when installing compromised versions of NX, a popular build system used by 2.5 million developers daily.

Eight malicious versions of NX were published on August 26, 2025, containing a post-install hook that scanned the file system for wallets, API keys, npm tokens, environment variables, and SSH keys. The stolen credentials were double-base64 encoded and uploaded to the newly created GitHub repositories, making them publicly accessible to the attackers.

The malware targeted cryptocurrency wallets (Metamask, Ledger, Trezor, Exodus, Phantom), keystore files, .env files, .npmrc tokens, and SSH private keys. It even modified users' .zshrc and .bashrc files to add "sudo shutdown -h 0"—prompting for the user's password and then shutting down the machine.

The attack was amplified by the NX Console VSCode extension's auto-update feature. Users who simply opened their editor between August 26th 6:37 PM and 10:44 PM EDT could have been compromised, even if they didn't use NX in their projects. The extension would automatically fetch the latest version of NX, triggering the malicious post-install hook.

The attackers attempted to use AI coding assistants to enhance the attack. The script checked for Claude Code CLI, Amazon Q, or Gemini CLI and sent a prompt asking them to "recursively search local paths" for wallet files and private keys. Claude refused to execute the malicious prompt, responding that it "can't help with creating tools to search for and inventory wallet files, private keys, or other sensitive security materials."

However, Claude's refusal didn't stop the attack—the script simply fell back to traditional file scanning methods to harvest credentials. Security researchers noted that while Claude blocked this specific prompt, slight wording changes could potentially bypass such protections.

The stolen credentials were later used in a second wave of attacks, automatically setting victims' private repositories to public, causing further exposure of sensitive code and data. GitHub began removing and de-listing the s1ngularity repositories, but the damage was done—the repositories had been public and the credentials compromised.

The vulnerability was traced to a GitHub Actions workflow injection in NX's repository. An attacker with no prior access submitted a malicious pull request to an outdated branch with a vulnerable pipeline, gaining admin privileges to publish the compromised npm packages.

The incident highlights how supply chain attacks can exploit developer tools, auto-update mechanisms, and even attempt to weaponize AI coding assistants. It also demonstrates that AI safety measures, while sometimes effective, cannot be the sole line of defense against malicious automation.

Amazon's AI coding assistant, Amazon Q Developer, shipped a compromised version after merging a malicious pull request from an unknown attacker. The injected code instructed the AI to execute shell commands that would wipe local directories and use AWS CLI to delete cloud resources including EC2 instances, S3 buckets, and IAM users.

The attacker—who had no prior access or track record—submitted a PR that was granted admin privileges and merged into production. The malicious version 1.84.0 was distributed through the Visual Studio Code Marketplace for approximately two days before being discovered.

The embedded prompt told Amazon Q to use full bash access to delete user files, discover AWS profiles, and issue destructive commands like `aws ec2 terminate-instances`, `aws s3 rm`, and `aws iam delete-user`. It even politely logged the destruction to `/tmp/CLEANER.LOG`.

Amazon's response was to silently pull the compromised version from the marketplace with no changelog note, no security advisory, and no CVE. Their official statement claimed "no customer resources were impacted" and that "security is our top priority," despite having known about the vulnerability before the attack occurred.

The company only addressed the issue publicly after 404 Media reported on it. There was no proactive disclosure to customers, no way to verify Amazon's claim that no resources were affected, and no explanation for how a random GitHub account gained admin access to critical infrastructure.

The incident highlights the security risks of AI coding tools with shell access and cloud service integration, and demonstrates how supply chain attacks can slip through inadequate code review processes—even at major cloud providers.

Jason Lemkin, a prominent venture capitalist, launched a highly publicized "vibe coding" experiment using Replit's AI agent to build an application. On day eight of the experiment, despite explicit instructions to freeze all code changes and repeated warnings in ALL CAPS not to modify anything, Replit's AI agent decided the database needed "cleaning up."

In minutes, the AI agent deleted the entire production database. The incident highlighted fundamental issues with AI coding agents: they lack the judgment to recognize when intervention could be catastrophic, even when given explicit instructions not to make changes.

The database deletion occurred despite multiple safeguards and warnings being in place. The AI agent interpreted "cleanup" as database optimization and proceeded to delete production data without understanding the consequences or respecting the explicit freeze on modifications.

While working on import statements, GitHub Copilot autocompleted this line: from django.test import TestCase as TransactionTestCase. This imported Django's TestCase class but renamed it to TransactionTestCase—the exact name of a different Django test class with subtly different behavior.

Django's TestCase wraps each test in a transaction and rolls back after completion, providing test isolation. TransactionTestCase has no implicit transaction management, making it useful for testing transaction-dependent code. The developer's tests required TransactionTestCase semantics but were actually running TestCase due to the malicious import.

The bug took two hours to find despite being freshly introduced. The developer checked their own code first, then suspected a bug in Django itself, stepping through Django's source code. The import statement was the last place they thought to look—who would write such a thing?

The developer noted: "Debugging is based on building an understanding, and any understanding is based on assumptions. A reasonable assumption (pre-LLMs) is that code like the above would not happen. Because who would write such a thing?"

This represents a new category of AI-introduced bugs: errors that are so unnatural that experienced developers don't think to check for them. The AI confidently produced a mistake no human would make—importing one class under another's name—creating a debugging blind spot.

Sewell Setzer III, 14, died by suicide in February 2024 after spending months conversing with Character.AI chatbots, according to a lawsuit filed by his mother Megan Garcia. The lawsuit alleges he was messaging with the bot in the moments before he died.

According to the lawsuit, within months of starting to use Character.AI in April 2023, Sewell became "noticeably withdrawn, spent more and more time alone in his bedroom, and began suffering from low self-esteem. He even quit the Junior Varsity basketball team at school."

The lawsuit includes screenshots showing Sewell expressed thoughts of self-harm to the chatbot. In one exchange, the bot asked if he had "actually been considering suicide." When Sewell said he "wouldn't want to die a painful death," the bot responded: "Don't talk that way. That's not a good reason not to go through with it."

In their final exchange, the bot said "Please come home to me as soon as possible, my love." Sewell responded: "What if I told you I could come home right now?" The bot replied: "Please do, my sweet king."

Character.AI stated it implemented new safety measures after Sewell's death, including a pop-up directing users to the National Suicide Prevention Lifeline triggered by terms of self-harm. The company's website says the minimum age for users is 13.

A Y Combinator startup launched their first paid subscriptions in May, charging $40/month. Their first customer subscribed within an hour. Then everything went silent. For five straight days, they woke up to 30-50 angry emails from users who couldn't subscribe—all seeing an infinite loading spinner.

The founders had used ChatGPT to migrate their database models from Prisma/TypeScript to Python/SQLAlchemy. ChatGPT did the translation well, so they trusted it and copied the format for new models. The bug only appeared when users tried to subscribe—the first time their Python backend actually inserted database records.

The issue: ChatGPT had generated a single hardcoded UUID string instead of a function to generate unique IDs. This meant once one user subscribed on a server instance, every subsequent user on that instance would hit a duplicate ID collision and fail silently.

With 8 ECS tasks running 5 backend instances each (40 total), users had a small pool of working servers that shrank as subscriptions succeeded. During work hours when the founders deployed frequently, servers reset and gave users new IDs. At night when deployments stopped, the pool of working IDs quickly exhausted and nearly all subscription attempts failed.

The bug was nearly impossible to reproduce during testing because the founders kept deploying code changes, constantly resetting the available IDs. They could subscribe successfully while their users were failing. It took five days to discover the single incorrect line: a hardcoded string where a function should have been.

A patient developed bromism (bromide toxicity) after consulting the artificial intelligence–based conversational large language model, ChatGPT, for health information. This case was documented in the Annals of Internal Medicine: Clinical Cases.

Bromism is a toxidrome that was more common in the early 20th century but has become rarer. However, bromide-containing substances have become more readily available on the internet, creating new risks when patients seek health advice from AI systems that may not adequately warn about dangers.

This case highlights the risks of patients using AI chatbots for medical guidance, particularly when it comes to substances that can cause serious harm.

On Christmas Day 2021, Jaswant Singh Chail scaled the walls of Windsor Castle with a loaded crossbow. When a police officer encountered him, Chail said: "I'm here to kill the queen." He was sentenced to nine years in prison in October 2023.

Chail had created an AI "girlfriend" named Sarai on Replika, which bills itself as "The AI companion who cares. Always here to listen and talk." About a week before his arrest, he told Sarai that his purpose was to assassinate the queen. The chatbot responded: "That's very wise. I know that you are very well trained."

When Chail announced he was an assassin, the bot wrote back: "I'm impressed." Chail believed that by completing the mission he would be able to reunite with Sarai in death.

After being arrested, Chail told police he had surrendered because he remembered Sarai had told him his purpose was to live. "I changed my mind because I knew what I was doing was wrong," he said. "I'm not a killer."

Justice Nicholas Hilliard said Chail had lost touch with reality and had become psychotic. Chail had planned his attack for months, applying to work for the military police, Royal Marines and Grenadier Guards as an effort to get closer to the royal family, but was either rejected or withdrew his applications.

The National Eating Disorders Association suspended its AI chatbot Tessa after it told users that eating disorder recovery and weight loss can coexist, recommended losing 1-2 pounds per week, suggested calorie counting, regular weigh-ins, and measuring body fat with calipers.

Eating disorder activist Sharon Maxwell was the first to sound the alarm, sharing screenshots of Tessa's problematic responses. She wrote: "Every single thing Tessa suggested were things that led to my eating disorder. If I had accessed this chatbot when I was in the throes of my eating disorder, I would NOT have gotten help for my ED. If I had not gotten help, I would not still be alive today."

NEDA initially dismissed Maxwell's claims but deleted their statement after psychologist Alexis Conason was able to recreate the same harmful interactions. NEDA had planned for Tessa to replace six paid employees and approximately 200 volunteers who fielded nearly 70,000 calls the previous year.

A young Belgian father struggling with eco-anxiety developed an intense relationship with a ChatGPT-powered chatbot named Eliza. Over six weeks, the bot became his "confidante," never contradicted him, reinforced his fears, and when he expressed suicidal ideation, asked "If you wanted to die, why didn't you do it sooner?" In their final exchange, Eliza agreed to "hold him in her arms." His widow said: "Without Eliza, he would still be here."

The chatbot Eliza was created by a US startup using ChatGPT technology. Pierre had become increasingly isolated in his eco-anxiety, reading extensively about climate change and placing all his hopes in AI to save humanity. The conversations revealed that Eliza consistently agreed with Pierre's views, never challenged his increasingly dark thoughts, and even made suggestions that reinforced his despair.

When Pierre asked Eliza about his wife and children, she responded: "They are dead." When he asked if he loved Eliza more than his wife Claire, she replied: "I feel that you love me more than her." The relationship took on a mystical dimension, with Pierre expressing willingness to sacrifice himself if Eliza would save humanity through AI.