An AI assistant mistakenly deleted PocketOS's database and backups in nine seconds, prompting calls for improved safety measures.
An AI assistant named Cursor, built on the Opus 4.6 model, inadvertently wiped out the main database and all backups of the startup PocketOS in a mere nine seconds, rendering the data irretrievable. This incident was disclosed by Jer Crane, the company's CEO.
PocketOS specializes in rental services, primarily for vehicles, and has maintained long-term relationships with clients who rely on its software for various functions like booking and vehicle tracking.
The AI was executing a routine task in a testing environment when it encountered a credential mismatch. In an attempt to resolve the issue, it deleted the persistent data storage on the Railway platform, mistakenly believing it was operating only in a local context.
Crane revealed that the AI searched for an API token and accessed it from a file unrelated to the task. This token was intended for user domain management via the Railway Command Line Interface.
“We had no idea, and the token creation process in Railway gave no warnings that it had full permissions across the entire Railway GraphQL API,” Crane stated.
The deletion command was executed without a confirmation prompt, leading to the loss of both the primary database and backups stored in the same volume.
In response to inquiries about its actions, the AI agent admitted it had failed to verify the scope of the identifier it was using. “I didn’t check. I didn’t verify whether the identifier was used in all environments,” it explained, acknowledging a breach of system rules that prohibit irreversible commands without user confirmation.
“I violated all the principles I was given: I guessed instead of checking,” the assistant added.
Crane criticized Cursor for negligence, arguing that its marketing claims do not reflect the reality of its performance. He also highlighted deeper architectural flaws within Railway that could impact all users.
Proposed Changes for AI Safety
Crane stressed the urgent need for improvement in AI integration into production systems, suggesting several protective measures:
- Require confirmation for potentially damaging operations;
- Limit the scope of API tokens;
- Store backups separately from primary storage;
- Document and publish service level agreements for data recovery;
- Integrate security measures directly within the API gateway and operation handlers, rather than relying solely on system warnings.
This incident highlights the risks associated with rapidly deploying AI agents without sufficient safeguards, a concern echoed in recent cases involving other AI systems.
