A study from Zhejiang University reveals a method to manipulate AI voice models using inaudible audio commands, achieving success rates up to 96%.
Research Highlights Inaudible Manipulation of AI Voice Models
A team of researchers from Zhejiang University has developed a technique called AudioHijack, which embeds imperceptible commands into audio signals, achieving a remarkable success rate of 79% to 96% in manipulating large audio-language models (LALMs). This groundbreaking study was presented at the 47th IEEE Symposium on Security and Privacy in San Francisco.
The method effectively targets both open-source and commercial voice AI systems, including those from Microsoft and Mistral, with traditional defenses managing to thwart only a small fraction of the attacks. The researchers are now investigating the potential to apply this technique to closed models from OpenAI and Anthropic, leveraging shared open-source audio components.
Details of the AudioHijack Technique
AudioHijack operates by modifying the numerical values in a digital audio waveform in a way that remains inaudible to human listeners while still influencing how AI models interpret the sound. Meng Chen, a Ph.D. candidate at Zhejiang University, explained that the training process for this signal takes only about half an hour and is context-agnostic, allowing it to be deployed at any time regardless of the user's spoken commands.
Unlike traditional prompt injection attacks that require altering user speech, this technique directly modifies the audio signal, embedding hidden commands within frequencies undetectable by humans. This unique approach poses significant challenges for defense mechanisms that typically focus on identifying suspicious text prompts.
Implications and Potential Risks
The researchers tested AudioHijack on 13 open-source AI voice models, finding it capable of causing them to deny requests, spread misinformation, insert harmful links, and execute unintended actions like web searches and sending emails with sensitive information. The method's effectiveness against commercial systems underscores the need for improved security measures in AI technologies.
Moreover, the study highlights various potential delivery channels for these attacks, including online videos, music clips, and voice messages, with evidence suggesting similar attacks could occur in real-time AI voice interactions. The researchers noted that monitoring a model's internal attention mechanisms could serve as a countermeasure, although attackers might adapt their strategies to maintain effectiveness even against such defenses.
“Single-point defenses face challenges against our attack because we found it very difficult for these models to differentiate between normal user intentions and our adversarial attack,” Chen added.
