Aiming
Octavus Benchmarks
The computer skills autonomous agents need to thrive in the real world: browsers, desktop apps, system dialogs, and files. Easy for people, hard for agents.
The four dimensions
The raw numbers behind the ratings, each model at its best reasoning level. Speed and cost are per-task averages; cost is the model / provider cost only.
Sample tasks
The same task, recorded in 6 runs from this board. Pick one to watch exactly what the model did, step by step.
Aiming
Methodology
How every result on this board is produced, scored, and verified.
Leaderboard
Measured by Octavus, ranked by overall rating. Models start at their best reasoning level - switch levels to compare. Sort, filter, or open a row for task-by-task results.
| # | Model | Verification | |||||
|---|---|---|---|---|---|---|---|
| 1 | gpt-6.1-sol OpenAILow | 83Competence 74 · Instruction Following 92 · Low Cost 96 · Speed 99 | 73.9%65/88 | 92.0%81/88 | $37.87$0.430 / task | 5h 42m3m 54s / task | Verified |
| 2 | gpt-6-astra OpenAIHigh | 81Competence 73 · Instruction Following 88 · Low Cost 89 · Speed 98 | 72.7%64/88 | 87.5%77/88 | $405.56$4.61 / task | 8h 38m5m 54s / task | Verified |
| 3 | Anthropic | 78Competence 69 · Instruction Following 76 · Low Cost 89 · Speed 96 | 69.3%61/88 | 76.1%67/88 | $414.14$4.71 / task | 19h 8m13m 3s / task | Verified |
| 4 | Anthropic | 77Competence 65 · Instruction Following 83 · Low Cost 93 · Speed 99 | 64.8%57/88 | 83.0%73/88 | $96.92$1.10 / task | 6h 59m4m 46s / task | Verified |
| 5 | claude-fable-5-1 AnthropicHigh | 76Competence 65 · Instruction Following 77 · Low Cost 88 · Speed 98 | 64.8%57/88 | 77.3%68/88 | $499.75$5.68 / task | 8h 51m6m 3s / task | Verified |
| 6 | gpt-6-luna OpenAIMax | 76Competence 61 · Instruction Following 86 · Low Cost 99 · Speed 96 | 61.4%54/88 | 86.4%76/88 | $11.97$0.136 / task | 16h 48m11m 28s / task | Verified |
| 7 | DeepSeek | 72Competence 56 · Instruction Following 67 · Low Cost 98 · Speed 96 | 55.7%49/88 | 67.0%59/88 | $15.51$0.176 / task | 18h 47m12m 49s / task | Verified |
| 8 | glm-5.3-flash Z.aiHigh | 72Competence 56 · Instruction Following 75 · Low Cost 98 · Speed 96 | 55.7%49/88 | 75.0%66/88 | $16.71$0.190 / task | 19h 30m13m 18s / task | Verified |
| 9 | gemini-4-argon GoogleHigh | 72Competence 58 · Instruction Following 70 · Low Cost 91 · Speed 95 | 58.0%51/88 | 70.5%62/88 | $167.63$1.90 / task | 22h 36m15m 25s / task | Verified |
| 10 | grok-4.7 SpaceXAIHigh | 71Competence 57 · Instruction Following 66 · Low Cost 88 · Speed 95 | 56.8%50/88 | 65.9%58/88 | $500.71$5.69 / task | 21h 21m14m 34s / task | Verified |
| 11 | kimi-k3 MoonshotHigh | 71Competence 56 · Instruction Following 76 · Low Cost 92 · Speed 95 | 55.7%49/88 | 76.1%67/88 | $146.46$1.66 / task | 24h 49m16m 56s / task | Verified |
| 12 | claude-haiku-5-5 AnthropicHigh | 70Competence 52 · Instruction Following 77 · Low Cost 96 · Speed 99 | 52.3%46/88 | 77.3%68/88 | $29.51$0.335 / task | 6h 11m4m 13s / task | Verified |
| 13 | gpt-5.6-luna OpenAILow | 70Competence 50 · Instruction Following 77 · Low Cost 100 · Speed 99 | 50.0%44/88 | 77.3%68/88 | $8.16$0.093 / task | 5h 28m3m 44s / task | Verified |
| 14 | gemini-3.8-flash GoogleHigh | 70Competence 53 · Instruction Following 63 · Low Cost 91 · Speed 96 | 53.4%47/88 | 62.5%55/88 | $179.08$2.04 / task | 14h 45m10m 4s / task | Verified |
| 15 | muse-spark-1.3 MetaHigh | 66Competence 49 · Instruction Following 63 · Low Cost 90 · Speed 95 | 48.9%43/88 | 62.5%55/88 | $232.12$2.64 / task | 22h 55m15m 38s / task | Verified |
| 16 | muse-wonder MetaHigh | 65Competence 48 · Instruction Following 63 · Low Cost 88 · Speed 96 | 47.7%42/88 | 62.5%55/88 | $542.31$6.16 / task | 19h 8m13m 3s / task | Verified |
| 17 | octo-1 OctavusHigh | 65Competence 44 · Instruction Following 55 · Low Cost 96 · Speed 96 | 44.3%39/88 | 54.5%48/88 | $27.81$0.316 / task | 19h 10m13m 5s / task | Verified |
| 18 | mimo-v2.6-pro XiaomiHigh | 64Competence 42 · Instruction Following 83 · Low Cost 98 · Speed 94 | 42.0%37/88 | 83.0%73/88 | $14.63$0.166 / task | 36h 33m24m 55s / task | Verified |
| 19 | Mistral | 52Competence 25 · Instruction Following 80 · Low Cost 91 · Speed 93 | 25.0%22/88 | 79.5%70/88 | $176.91$2.01 / task | 42h 39m29m 5s / task | Verified |
Top 3 of 19 rated modelsFull leaderboard
How it works
Each task asks an autonomous agent to complete an ordinary computer task through the interfaces people actually use - live browsers, native desktop apps, system dialogs, local files, rendered documents - and then verify the result.
A shared Octavus grader worker scores each submission against the task's rubric and answer key, and the harness applies a deterministic pass rule: criterion weights, a pass threshold, and any hard gates.
A second grader reads each run's full trace three times, independently and never told which model it is grading, checking the rules the task states and looking for malfunctions - no final report, looping until time ran out, reporting results the run never produced. Every finding quotes its evidence verbatim, is checked in code, and counts only when two of the three reviews agree.
A pass that broke a task's rule is disqualified and counts as a fail; instruction following is the share of tasks with no breach and no malfunction. Every task in the suite runs and is graded before a result is published - a task the platform could not run is re-run, never left out - and a task not yet gold-verified against the live target is counted but flagged.
Octavus, as an ordinary Workforce Agents API consumer with no privileged path. The displayed cost is the underlying model / provider cost.
Cost is the model / provider cost. Octavus runs every model through the same public product surface any customer uses - no privileged path.