Nvidia's Cosmos 3 Nano model accuracy improved from 54.41% to 93.35% in a day, using AI agent Codex for fine-tuning with LoRA.
Nvidia has successfully boosted the accuracy of its Cosmos 3 Nano model on a multiple-choice test from 54.41% to 93.35% in under a day, leveraging AI agent Codex for fine-tuning.
This significant improvement utilized the Woven Traffic Safety dataset from Woven by Toyota, which comprises videos depicting traffic scenarios along with corresponding questions and answer choices. The training involved over 8,000 examples.
Initially, the Cosmos 3 Nano model performed at 54.41% accuracy. To enhance its performance, Codex was tasked with evaluating the base model and applying fine-tuning through the Low-Rank Adaptation (LoRA) technique.
Codex utilized the specialized instruction Cosmos-reason, verified dataset annotations, and identified a missing frame rate parameter. After correcting this, it proceeded to load model weights, configure settings, and initiate the training container.
LoRA focuses on training a limited set of additional adapters rather than adjusting all model parameters, which Nvidia claims requires about seven times fewer GPU hours compared to a complete retraining of Cosmos 3 Nano. The first LoRA run lasted around 30 minutes on eight NVIDIA A100 accelerators with 80 GB of memory, resulting in an accuracy of 87.14%.
In a subsequent step, developers employed TAO AutoML to optimize various settings, including learning rate and batch size. This process involved 43 parallel trials and achieved a peak accuracy of 93.35% through Bayesian optimization, taking 19.5 hours across multiple A100 nodes in Oracle Cloud Infrastructure.
Throughout the experiment, Codex was engaged in various tasks, such as analyzing results, selecting workflows, and launching containers, though its autonomy was limited to following pre-defined instructions.
It's important to note that the 93.35% accuracy pertains to the validation set of a single research dataset, and does not evaluate the model's safety for autonomous driving or its capability for real-time decision-making.
