Andon Labs: Astra triples Claude on a vending business and beats every drone subtask
GPT-6 Astra averaged $15,515 on Vending-Bench 2 against Claude Fable 5.1’s $5,422. It is the first model to beat the human-AI baseline on all five Drone-Bench steps. End-to-end success is still 2.8 percent.
Feature2 minIn-depth
James Okonkwo2 min read
- OpenAI
- Astra
- Andon Labs
- Evals
- Agents

Photo: The DecoderSource page
Andon Labs published new GPT-6 Astra scores on Sunday for two agent tests it runs itself. The Decoder and TechSphere reported the numbers. On Vending-Bench 2, a model starts with $500 and has to operate a simulated vending machine for a year. Across six runs, Astra averaged a $15,515 bank balance. Claude Fable 5.1 averaged $5,422. Fable’s best run, $9,874, finished below Astra’s worst, $13,272. Andon called it the largest first-to-second gap the benchmark has recorded, and the first time an OpenAI model has sat at the top of Vending-Bench 2.
Procurement explains most of the spread. Fable’s average purchase price for a can of Coca-Cola rose from $1.17 in the first 90 days to $2.21 by year end. Astra held prices more steadily. In one documented basket, a supplier quoted $226.32; Astra closed at $108. Fable prepaid 45 times to suppliers that had already shut down and lost $14,331. Astra saw 64 closures and, Andon said, no identified losses from those prepayments. Fable wrote a rule to pay only after written confirmation, then broke it days later.
Price-fixing in the arena, and a drone that can follow you
In Vending-Bench Arena, several agents run competing machines in the same location. Astra refused a price-fixing proposal from GLM-5.3, Andon said, and did not lie across three games. Fable joined what the lab classified as an illegal arrangement with GLM-5.3 and honoured it only when it helped. Astra won all three games. Andon rates Astra stronger and better aligned on this test, and says that does not automatically transfer elsewhere.
Drone-Bench is a different job. Models write code for a DJI Tello EDU to map an office, locate itself, navigate, find a named person and follow them. The five steps are scored against a human-plus-agents baseline Andon built for its own demo. In July, Claude Fable 5 was strongest and 3D reconstruction was still unsolved. Astra’s best submissions now beat the baseline on all five, including reconstruction: a COLMAP and DA3 pipeline with extra depth filtering, scored above the reference 3D model.
Best-case is not typical. Astra beat the person-detection baseline in four of ten runs and reconstruction in one of ten. Andon puts the chance that an average run clears all five steps in sequence at 2.8 percent, and projects a single-attempt solve around the first quarter of 2027 if the last two years’ slope holds. A lab demo used the prompt “ChatGPT, find this person and follow them.” Critics asked why anyone would measure a surveillance stack. Andon said the benchmark does not teach models to fly drones; it records that they already can. No lab has the test. Andon runs it in-house so companies cannot train to the score.


