Summary
- A team from Multiverse Computing unveiled a new approach called Quantization-Aware Healing on the Hugging Face blog on August 25.
- They successfully reduced OpenAI's GPT-OSS model from 120 billion parameters to 60 billion, while also compressing its memory to 4-bit, and the downsized model excelled in 7 out of 9 evaluations compared to the original.
- The key was to train the smaller model using the original’s knowledge instead of relying on a less capable intermediate version.
A group of researchers has created a more compact, cost-effective AI model that surprisingly outperforms its larger predecessor.
This phenomenon is unusual, akin to losing weight while gaining strength. According to the team at Multiverse Computing, this success indicates that traditional methods of reducing AI size may need reevaluation.
“For practitioners, the practical message is that in a distillation-based healing pipeline, the quantization step is not a cost to be minimized but an additional opportunity for teacher supervision, yielding a model that is simultaneously cheaper to serve, lighter in memory, and at least as accurate as its full-precision counterpart,” the researchers explained in their paper published on Friday.
AI model parameters can be likened to a vast array of dials—each representing a piece of knowledge. More dials signify a smarter model, but they also require more resources. OpenAI's GPT-OSS, with its 120 billion parameters, exemplifies this, consuming considerable memory and power.
To make AI more affordable, companies often reduce the number of parameters, which can lead to a loss in quality, similar to compressing a high-resolution image to a lower quality. Traditionally, excessive compression results in diminished performance. However, these researchers took a different approach.
They trimmed GPT-OSS down to 60 billion parameters and fit each into a compact 4-bit format, which typically would compromise the model's capabilities. Yet, they discovered a way to enhance its performance instead.
In nearly all comparative benchmarks, the smaller model outperformed its full-precision counterpart.
The Mistake of Copying
Typically, when creating a smaller model, developers rely on a “half-shrunk” version for corrections. However, this intermediate model is already an inferior copy of the original. Hence, the smaller model learns to replicate a flawed version, limiting its potential for improvement.
The researchers’ method, known as “Quantization-Aware Healing,” shifts the focus. It instructs the smaller model to learn from the original, uncompressed version instead of the flawed intermediate. As a result, the 60-billion-parameter model surpassed its higher-precision twin in 7 out of 9 tests. While the original 120 billion parameter model still performed better overall, the reduced version achieved results that exceeded expectations.
The implications of a smaller, more efficient model are significant, as AI typically demands substantial hardware resources. The new model requires about a quarter of the memory and half the parameters of the original, bridging the gap between large data center deployments and more accessible desktop or mobile applications.
This development is particularly beneficial for smaller labs and independent developers, as the model produces superior results while consuming less energy.
Additionally, the team has made the Hypernova-60B model available for free as open weights on Hugging Face, enabling anyone to download and utilize it. This follows a trend of open-source models achieving remarkable performance, such as the enigmatic Ox Alpha model recently outperforming a Claude system, and the excitement surrounding Alibaba’s Qwen 3.8 Flash Next and various fine-tuned models that enhance smaller models using insights from larger ones like Fable or Claude Opus.
However, caution is advised; the proprietary shrink tool used to create the 60B model is not fully open-source, and testing was limited to GPT-OSS, excluding families like Llama, Qwen, or Mistral.