Huawei's Claw-Anything benchmark reveals significant performance issues in AI agents, particularly OpenAI's GPT-5.5, which scored only 34.5%.
Huawei Develops Claw-Anything Benchmark
A collaboration between Huawei Technologies and three academic institutions has led to the creation of the Claw-Anything benchmark, aimed at evaluating AI agents in personal-assistant roles. This benchmark is designed to test the capabilities of AI in managing various digital tasks over an extended period, revealing significant performance gaps.
Low Performance Scores Highlight Flaws
OpenAI’s latest model, GPT-5.5, achieved a disappointing pass@1 score of just 34.5% on this benchmark. This result raises concerns about the effectiveness of traditional evaluation methods for AI agents, as it falls short of expectations set by other benchmarks.
The Claw-Anything benchmark assesses AI capabilities across three dimensions: managing long-term event streams simulating over three months of user activity, handling an average of 10.1 interdependent backend services per task, and operating across both CLI Linux and GUI Android environments. Each task presents a context of approximately 191,700 words, significantly larger than the typical benchmark's range of 1,700 to 12,000 words, illustrating the disconnect between real-world applications and conventional testing scenarios.
Proactive vs. Reactive Assistance
The benchmark not only measures task completion but also evaluates proactive assistance, where the AI identifies user needs without explicit prompts. Results showed a stark difference in performance, with agents scoring 25.9% on reactive tasks and only 6.7% on proactive tasks, indicating a substantial limitation in current AI capabilities.
Rethinking AI Evaluation Criteria
The researchers argue that traditional benchmarks often oversimplify the challenges faced by AI agents, treating them as task solvers in ideal conditions. In contrast, Claw-Anything simulates the complexities of real life, where agents must navigate irrelevant events and conflicting information. The study emphasizes the critical role of multi-service dependencies, as excluding necessary tools for cross-service tasks resulted in near-zero success rates.
Despite the challenges highlighted, the research team successfully fine-tuned an open-weight model, Qwen3.5-27B, using the benchmark's training data, resulting in a 23.7% improvement in performance. This advancement positions it favorably against other closed-source models in the field.
The dataset and code for the Claw-Anything benchmark are now available on Hugging Face and GitHub, contributing to ongoing efforts to enhance AI performance in real-world applications.
