Claude Opus 4.8 shows improved performance in math and coding but struggles with high token consumption and limited advancements in creative writing.
Overview
- Opus 4.8 excelled in mathematical tasks, outperforming previous models in a one-prompt game.
- One coding task exhausted the entire Pro token allocation, limiting its practicality for larger projects.
- Creative writing performance did not significantly improve from version 4.7.
Anthropic has launched Claude Opus 4.8, just six weeks after the debut of version 4.7. While the pricing remains unchanged at $5 per million input tokens and $25 per million output tokens, the new version shows improvements in performance and safety metrics.
The model underwent a series of evaluations covering creative writing, coding, mathematics, logical reasoning, narrative analysis, and long-context recall, with results compared against its predecessor and competing models from China.
Overall, while Opus 4.8 has made strides in areas like mathematics and coding, it has not shown much progress in creative writing and suffers from high token consumption that could limit its usability.
Creative Writing Performance
In a creative writing exercise, Opus 4.8 crafted a time-travel narrative set in the Orinoco delta in the year 1000, featuring a character named José Lanz. The story's vivid descriptions, such as the delta's "green in a way 2150 had forgotten green could be," are commendable. However, despite its detailed execution, the writing lacks the fluidity and vibrancy seen in some competing models, and improvements over version 4.7 are marginal.
While the narrative structure is competent, it may benefit from a more dynamic approach through multi-shot prompting, as a single attempt did not yield a substantial upgrade in quality.
Coding Evaluation
In coding assessments, Opus 4.8 successfully developed a typing-zombie game called Typing Dead, demonstrating impressive game mechanics and design. The model effectively identified and corrected bugs during the development process. However, the significant drawback emerged when a single coding prompt consumed the entire token allocation for Pro users, making it impractical for larger coding projects.
Mathematics Capability
In mathematical evaluations, Opus 4.8 tackled a complex problem involving a degree-19 polynomial and accurately computed p(19) = 1,876,572,071,974,094,803,391,179. This performance marks a notable improvement over version 4.7, which struggled with similar tasks.
Logical Reasoning
For logical reasoning, Opus 4.8 successfully identified the contradiction in a question about marrying a widow's sister, demonstrating solid reasoning skills. It provided a thorough analysis of the legal context, aligning with expectations set by competing models.
Non-Mathematical Reasoning
However, in a mystery scenario requiring careful deduction, Opus 4.8 failed to identify the correct culprit, showcasing a risk associated with LLMs: they can present convincing but incorrect conclusions. This raises concerns for users relying on AI for accurate information.
Context Management
In tests involving context management, the 300K-token version failed to operate effectively, undermining claims of handling larger contexts. The 85K version performed adequately in identifying specific content but ultimately declined to answer due to safety protocols, indicating a unique operational failure.
Conclusion
The evaluations suggest that while Opus 4.8 enhances certain strengths, it may also diminish some weaknesses. This indicates a targeted approach towards developers, particularly those with financial means. The high token consumption issue remains a significant concern, potentially driving users to consider more cost-effective alternatives.
Despite these challenges, Anthropic's strategy appears to be succeeding, with indications of a forthcoming public offering and a valuation nearing $1 trillion.
