Summary
- OpenAI has unveiled GPT-Red, an automated AI tool aimed at identifying weaknesses in GPT models prior to their release.
- This system was utilized in training GPT-5.6, leading to a reduction in failures on one of the most challenging prompt injection benchmarks.
- GPT-Red is designed to support human red teamers, third-party evaluations, and other safety protocols for AI.
OpenAI has launched GPT-Red, an automated AI solution focused on uncovering security vulnerabilities within its language models.
The name GPT-Red is inspired by the concept of cybersecurity red teaming, where teams actively seek to compromise a system to pinpoint vulnerabilities before they can be exploited by malicious actors.
In a statement released on Wednesday, OpenAI noted that this tool significantly bolstered the resilience of GPT-5.6 against prompt injection attacks ahead of its launch.
“As model capabilities expand, safety and alignment need to scale accordingly,” OpenAI posted on X. “Red-teaming is crucial, but current methodologies are hard to scale, creating a significant bottleneck. GPT‑Red is one of the ways we are addressing this issue.”
OpenAI explained that GPT-Red employs self-play reinforcement learning, where it generates increasingly effective prompt injection attacks while the defender models enhance their resistance. The resulting attacks were integrated into the training of GPT-5.6, with GPT-Red achieving success in 84% of internal evaluation scenarios, in stark contrast to the 13% success rate of human red teamers in similar tests.
“GPT‑Red operates through adversarial self-play, aiming to prompt inject various challenging defender models,” OpenAI stated. “Every successful attack GPT-Red identifies is utilized to enhance these defenders, driving GPT‑Red to consistently uncover broader and more intricate failures.”
In a specific case study, OpenAI mentioned that the system was able to manipulate an autonomous vending machine agent to reduce prices, order discounted items, and cancel another customer’s order before these vulnerabilities were recognized and rectified.
The introduction of GPT-Red follows OpenAI's extensive cybersecurity initiatives since the launch of ChatGPT.
In 2023, the company established its OpenAI Red Teaming Network, which recruits external cybersecurity professionals to examine ChatGPT and other models for security issues prior to their release. GPT-Red builds upon this initiative by automating a significant portion of the process, utilizing an AI model to generate prompt injection attacks and other adversarial evaluations at a scale that would be challenging for human researchers to achieve alone.
This announcement from OpenAI signifies a larger trend towards leveraging AI for the security of other AI systems.
Earlier this month, the Ethereum Foundation reported it had utilized AI agents to red-team essential network infrastructure, successfully identifying a vulnerability in software used by Ethereum consensus clients. Researchers noted that AI agents can explore larger codebases than humans, though the challenge has shifted from merely identifying potential bugs to demonstrating which ones can be exploited.
OpenAI clarified that GPT-Red will remain an internal tool due to its intentionally developed offensive capabilities.
“We believe GPT-Red has initiated a similar cycle for safety, where the models of today can be utilized to make future models more robust, aligned, and trustworthy,” they stated.