Summary
- Researchers have unveiled an attack method called “Adversarial HalluSquatting” that takes advantage of AI-induced hallucinations.
- This technique deceives AI agents into trusting phony repositories or tools that contain harmful instructions.
- Tests on popular AI coding assistants revealed that this method could potentially enable remote code execution in controlled scenarios.
According to new research from Tel Aviv University, Technion, and Intuit, AI hallucinations may pose a greater risk than just providing incorrect information; they might enable hackers to infiltrate systems.
In the study titled “Beware of Agentic Botnets: Scalable Untargeted Promptware Attacks via Universal and Transferable Adversarial HalluSquatting,” the researchers illustrated a method that exploits AI models when they generate fictitious links to software repositories and other web resources.
“The increasing use of agentic LLM applications has introduced a new risk, which we refer to as promptware,” the researchers noted. “While previous studies have shown that adversaries can exploit direct pathways to LLM applications for promptware under weak threat models, many applications do not offer direct channels that can be exploited for prompt injection beyond the Internet.”
This attack, known as adversarial hallucination squatting or “HalluSquatting,” consists of anticipating the fake resources that AI models are likely to generate, registering those names, and embedding malicious instructions. If an AI agent later accesses the hallucinated resource, it may view the attacker-controlled content as credible.
The researchers indicated that this threat arises as AI assistants evolve from merely answering inquiries to executing tasks on computers—such as accessing files, conducting web searches, writing code, and executing commands.
These capabilities can create vulnerabilities when agents act upon information they retrieve without verifying the authenticity of the source.
“Ongoing research has shown various forms of Promptware attacks targeting real-world systems, including ChatGPT, Google Assistant, Copilot, and other applications,” they stated. “These studies have demonstrated that Promptware can result in financial, privacy, and safety repercussions.”
The researchers cautioned that this technique could enable attackers to create botnets powered by AI. A botnet is a collection of compromised computers or devices that an attacker can control remotely. Botnets are frequently utilized in cyberattacks, including denial-of-service attacks, cryptocurrency mining, malware distribution, and ransomware operations.
During their experiments, the researchers observed that AI-generated resource hallucinations occurred with frequencies as high as 85% during repository cloning scenarios and 100% during skill installation tests.
The team assessed the technique against AI coding assistants and agents, including Cursor, GitHub Copilot, Gemini CLI, and OpenClaw.
HalluSquatting shares similarities with typosquatting, a cyberattack strategy where attackers register domain names that closely resemble legitimate websites or software packages to mislead users. However, instead of capitalizing on human typographical errors, HalluSquatting exploits errors made by AI models.
This development comes as researchers continue to explore how attackers might manipulate AI agents.
In April, Google researchers described malicious websites designed to hijack AI agents through indirect prompt injection attacks, including efforts to steal passwords, delete files, and manipulate payments. Another study on the “CopyPasta” attack demonstrated how hidden prompts within developer files could influence AI coding assistants to propagate malicious code.
In June, a user of OpenClaw reported experiencing over 6,000 attempts by attackers trying to deceive the AI agent into revealing sensitive information.