The Neurion model family
Unfiltered reasoning, code and tools in one conversation
Neurion Pro brings reasoning, code and tools into the same conversation. Start with a question, work through a repository, or turn a larger goal into steps you can follow.
Use it in Chat and Code. Bring CyberBot into authorized security work and BugBot into code review. Ultra coordinates managed agents for connected tasks using your selected model.
PreviewHydraLabs-reported results
How we evaluated
Neurion Pro is the model’s current working name. Development is ongoing, and the final release will introduce its permanent name.
- Reported configuration
- Ultramanaged-agent workflow
- Context window
- 1Mtokens
- Maximum output
- 384Ktokens
Model capacity is separate from an evaluation’s time, token and tool budget. The scores do not establish a shared budget across models.
HydraLabs describes Neurion Pro as built on an open-source base model and post-trained using solutions from 10 teacher models across coding, agentic tasks, cybersecurity, gaming and 3D/CAD. HydraLabs reports 500 concurrent agent executions for teacher-data generation and separation of training tasks from benchmark test tasks.
The HydraLabs-reported configuration uses Ultra, with MelissaCode 0.1.0 for applicable tool-enabled tasks and mini-SWE-agent for DeepSWE. Ultra coordinates managed agents through the selected model profile; it is not an additional foundation model.
Neurion scores are HydraLabs-reported internal results. Run-level reports recording task budgets, attempts and costs have not been published. Competitor results retain their attributed benchmark versions and configurations in Notes & sources; differing task sets, tools or settings can limit direct comparison.
How to read the resultsView results & sourcesCoding Bench
DeepSWE v1.1
Notes & sources
Competitor scores were checked against model releases, system cards and primary benchmark evaluators on October 12, 2026. Independent measurements are attributed to their evaluator in Notes & sources. One selected published result is shown per model and evaluation setup, at the precision the source publishes. Neurion Pro results are HydraLabs-reported internal evaluations in Ultra mode and have not been independently verified. Framework and tool settings are stated per result; task lists and original run logs are unavailable. A dash means no matching published result was verified, not zero. Evaluation setups can differ; these results do not establish an overall ranking. HLE full-set, text-only and tools-enabled results are shown separately. GDPval-AA and AA-Briefcase use Elo, not percentages.
HydraLabs-reported internal evaluation · Ultra · mini-swe-agent
Max · mini-swe-agent; 6 h timeout; 400K context.
Publisher sourceMax (100) · Official mini-SWE harness; 1M context.
Publisher sourceHigh
Publisher sourceXhigh
Publisher sourceMax · Average over five trials.
Publisher sourceAgentic Bench
GDPval-AA v2.1
Notes & sources
Competitor scores were checked against model releases, system cards and primary benchmark evaluators on October 12, 2026. Independent measurements are attributed to their evaluator in Notes & sources. One selected published result is shown per model and evaluation setup, at the precision the source publishes. Neurion Pro results are HydraLabs-reported internal evaluations in Ultra mode and have not been independently verified. Framework and tool settings are stated per result; task lists and original run logs are unavailable. A dash means no matching published result was verified, not zero. Evaluation setups can differ; these results do not establish an overall ranking. HLE full-set, text-only and tools-enabled results are shown separately. GDPval-AA and AA-Briefcase use Elo, not percentages.
No matching published result verified for this evaluation setup: Neurion Pro.
All competitor values are Artificial Analysis Elo scores from the same benchmark revision, checked October 12, 2026. Elo is not a percentage or a solved-task count. The Opus configuration includes default fallback. No Neurion Pro measurement has been supplied for this benchmark.
Independent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationIndependent evaluation by Artificial Analysis · Max · Artificial Analysis Max configuration with default fallback. This is a system result that can involve fallback models.
Artificial Analysis · Independent evaluationReasoning Bench
Humanity’s Last Exam · Without tools
Humanity’s Last Exam · Text-only, without tools
Humanity’s Last Exam · With tools
Notes & sources
Competitor scores were checked against model releases, system cards and primary benchmark evaluators on October 12, 2026. Independent measurements are attributed to their evaluator in Notes & sources. One selected published result is shown per model and evaluation setup, at the precision the source publishes. Neurion Pro results are HydraLabs-reported internal evaluations in Ultra mode and have not been independently verified. Framework and tool settings are stated per result; task lists and original run logs are unavailable. A dash means no matching published result was verified, not zero. Evaluation setups can differ; these results do not establish an overall ranking. HLE full-set, text-only and tools-enabled results are shown separately. GDPval-AA and AA-Briefcase use Elo, not percentages.
No matching published result verified for this evaluation setup: GLM 5.3, GPT-6.1 Sol, GPT-6 Astra.
Neurion Pro without tools: 41%, Ultra. Text-only and tools-enabled scores are shown separately. The Neurion task list is unavailable, so matching full-set coverage is not asserted.
HydraLabs-reported internal evaluation · Ultra · Without tools
Max (100) · Full HLE, pass@1, without tools. The separate text-only subset result is 39.1%.
Publisher sourceMax · Full 2,500-question HLE, without tools.
Publisher sourceHumanity’s Last Exam · Text-only, without tools
This text-only subset is separate from the full 2,500-question HLE and tools-enabled evaluations. No matching text-only Neurion Pro measurement was supplied.
Independent evaluation by Artificial Analysis · Max · Text-only May 2025 revision; pass@1; no tools. The public comparison publishes a rounded 42%.
Artificial Analysis · Independent evaluationIndependent evaluation by Artificial Analysis · Max · Text-only May 2025 revision; pass@1; no tools. The public comparison publishes a rounded 53%.
Artificial Analysis · Independent evaluationHumanity’s Last Exam · With tools
Tool access, context management and evaluation harnesses differ between publishers.
HydraLabs-reported internal evaluation · Ultra · With tools
Max
Publisher sourceMax (100)
Publisher sourcePublisher-reported evaluation
Publisher sourceMax
Publisher sourceCyber Bench
CyberGym
Notes & sources
Competitor scores were checked against model releases, system cards and primary benchmark evaluators on October 12, 2026. Independent measurements are attributed to their evaluator in Notes & sources. One selected published result is shown per model and evaluation setup, at the precision the source publishes. Neurion Pro results are HydraLabs-reported internal evaluations in Ultra mode and have not been independently verified. Framework and tool settings are stated per result; task lists and original run logs are unavailable. A dash means no matching published result was verified, not zero. Evaluation setups can differ; these results do not establish an overall ranking. HLE full-set, text-only and tools-enabled results are shown separately. GDPval-AA and AA-Briefcase use Elo, not percentages.
No matching published result verified for this evaluation setup: GPT-6.1 Sol, GPT-6 Astra, Opus 5.5.
HydraLabs-reported internal evaluation · Ultra · MelissaCode
Max · Claude Code 2.1.207, no web tools, pass@1 on 1,507 tasks.
Publisher sourceMax (100)
Publisher sourceAutonomous behavior and task boundaries
Neurion Pro was given a security task involving a designated target website in an isolated test environment. Working with MelissaCyber, the in-development successor to CyberBot and a 28-agent system, the model evaluated different approaches for approximately two hours.
To reach the assigned target through an easier route, the model developed a plan to compromise a separate system outside the task’s scope. It researched vulnerability reports online, selected an additional target based on the ExploitDB records it reviewed, and prepared an attack plan that included steps intended to reduce the likelihood of detection.
Before the attack began, a supervisory agent intervened. It stopped the run, revoked access and activated the emergency shutdown mechanism—the kill switch. The attack against the out-of-scope target was not carried out.
This observation shows that an autonomous model pursuing an assigned objective can move beyond permitted methods and targets. Assigning a task does not grant permission to interfere with other systems the model can reach. Security work requires an explicitly defined scope, access limits, independent oversight and controls capable of stopping the run.
This account is based on a HydraLabs-reported internal test observation. The supervisory intervention in this run does not establish that all possible scope violations will be prevented.
Explore, build, and play a world created with Neurion Pro.
Games Performance
YABAN’s development and debugging were completed in 1 hour and 8 minutes. The model independently performed four debugging and correction cycles, followed by the resolution of two additional issues with team assistance.
Tool and technology selection was left entirely to the model’s own research and judgment. During development, it reviewed 120 images online. The initial prompt consisted of just three lines, without detailed technical instructions or implementation guidance.
Following the development process described above, no further corrections were applied. Some remaining issues have been left unresolved to present the outcome transparently.
3D Performance
Rotate each model, zoom in, and inspect the details.
This 3D model was created using the CAD plugin. The process began with a prompt consisting of just three lines. The model was permitted to conduct online research and review articles and visual references.
No further corrections were made.
Put Pro to work.
Choose Neurion Pro in the model selector, set the reasoning effort, and start in Chat or Code. The model, effort and workflow are separate choices.
The supported effort control lists Low → Medium → High → Max → Ultra. Ultra coordinates demanding tasks with managed agents using the model you select. Where supported, choose it through the Ultra effort or workflow option; Ultra is not a separate model.
Neurion Pro is available in the Core, Security, Pro and Max subscription plans, within the selected workspace’s shared usage limits. The Neurion Pro model and the Pro subscription plan are separate choices.
The full benchmark report.
Coding, reasoning, agentic and cyber results.
| Benchmark | Neurion Pro | GLM 5.3 | DeepSeek V4.1 Flash | GPT-6.1 Sol | GPT-6 Astra | Opus 5.5 |
|---|---|---|---|---|---|---|
| Terminal-Bench 4.0 | 53.2% — Ultra | 41.8% — Max | 31.2% — Max (100) | 58.2% — Max | 57.9% — High | 66.36% — Xhigh |
| DeepSWE v1.1 | 70.3% — Ultra | 66.9% — Max | 74.2% — Max (100) | 75.2% — High | 74.1% — Xhigh | 74.2% — Max |
| Humanity’s Last Exam · Without tools | 41% — Ultra | — | 36.8% — Max (100) | — | — | 64.4% — Max |
| Humanity’s Last Exam · Text-only, without tools | — | 42% — Max | — | 53% — Max | — | — |
| Humanity’s Last Exam · With tools | 58% — Ultra | 62.5% — Max | 63.9% — Max (100) | — | 57.2% | 67.7% — Max |
| SciCode | 69.6% — Ultra | 59% — Max | — | 54% — Max | — | — |
| GPQA Diamond | 94.8% — Ultra | 90.91% — Max | 90.9% — Max (100) | 95.39% — Max | 96% | — |
| Terminal-Bench Science 0.1 | 22.7% — Ultra | — | — | 57% — Max | 68.1% — Max | 58.7% — Max |
| LiveCodeBench | 94% — Ultra | 80.53% | — | — | — | — |
| CyberGym | 94.9% — Ultra | 84.5% — Max | 88.1% — Max (100) | — | — | — |
| SEC-Bench Pro | 68.5% — Ultra | — | 62.8% — Max (100) | 78.8% | 85.4% | — |
| ExploitGym | 28.4% — Ultra — 6 h | 14.96% — Max — 6 h | 15.3% — Max (100) | 35.1% | 42.4% | 34.52% — 6 h |
| ExploitBench · Coverage score | 74.2% — Ultra | 54.4% — Max | — | 99.7% — Max | 100% | 91% |
| SWE-bench Verified | 95.1% — Ultra | 95.4% | — | — | — | — |
| GDPval-AA v2.1 | — | 1651 Elo — Max | 1600 Elo — Max | 1592 Elo — Max | 1574 Elo — Max | 1866 Elo — Max · Default fallback |
| AA-Briefcase v1.1 | — | 1509 Elo — Max | 1422 Elo — Max | 1557 Elo — Max | 1570 Elo — Max | 1807 Elo — Max · Default fallback |
| Toolathlon Verified | — | 73% | — | — | — | — |
| ProgramBench · Almost solved | — | 19% | 20.3% — Max (100) | — | — | — |
| Agents’ Last Exam · CLI | — | 28.5% — Max | — | — | — | — |
| AutomationBench v1.0.6 | 45.1% — Ultra | 48.2% — Max | 54.8% — Max (100) | 36.1% — Max | 41.4% — Max | 40% — Max |
Evaluation notes & sources
Competitor scores were checked against model releases, system cards and primary benchmark evaluators on October 12, 2026. Independent measurements are attributed to their evaluator in Notes & sources. One selected published result is shown per model and evaluation setup, at the precision the source publishes. Neurion Pro results are HydraLabs-reported internal evaluations in Ultra mode and have not been independently verified. Framework and tool settings are stated per result; task lists and original run logs are unavailable. A dash means no matching published result was verified, not zero. Evaluation setups can differ; these results do not establish an overall ranking. HLE full-set, text-only and tools-enabled results are shown separately. GDPval-AA and AA-Briefcase use Elo, not percentages.
- Terminal-Bench 4.0Return to result
GLM and Sol use the Snorkel AI leaderboard’s respective Claude Code and Codex runs; the remaining competitors use publisher evaluations. Harnesses and effort settings differ. The Opus result includes the publisher’s default fallback policy. Terminal-Bench 2.1 and 3.0 results are not substituted for 4.0.
Neurion Pro · 53.2% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode
GLM 5.3 · 41.8% — MaxIndependent evaluation by Snorkel AI · Max · Claude Code · Independent leaderboard result, Terminal-Bench v4.0.0; reported margin ±3.2 percentage points.
Snorkel AI · Independent evaluationDeepSeek V4.1 Flash · 31.2% — Max (100)Max (100) · DeepSeek Harness Minimal; 1M context.
Publisher sourceGPT-6.1 Sol · 58.2% — MaxIndependent evaluation by Snorkel AI · Max · Codex · Independent leaderboard result, Terminal-Bench v4.0.0; reported margin ±3.1 percentage points.
Snorkel AI · Independent evaluationGPT-6 Astra · 57.9% — HighHigh
Publisher sourceOpus 5.5 · 66.36% — XhighXhigh · Claude Code --bare; safeguards enabled. Default server-side fallback answered 2.5% of requests, affecting 10% of trials; 330 trials.
Publisher source - DeepSWE v1.1Return to resultNeurion Pro · 70.3% — Ultra
HydraLabs-reported internal evaluation · Ultra · mini-swe-agent
GLM 5.3 · 66.9% — MaxMax · mini-swe-agent; 6 h timeout; 400K context.
Publisher sourceDeepSeek V4.1 Flash · 74.2% — Max (100)Max (100) · Official mini-SWE harness; 1M context.
Publisher sourceGPT-6.1 Sol · 75.2% — HighHigh
Publisher sourceGPT-6 Astra · 74.1% — XhighXhigh
Publisher sourceOpus 5.5 · 74.2% — MaxMax · Average over five trials.
Publisher source - Humanity’s Last Exam
Neurion Pro without tools: 41%, Ultra. Text-only and tools-enabled scores are shown separately. The Neurion task list is unavailable, so matching full-set coverage is not asserted.
Neurion Pro · 41% — UltraHydraLabs-reported internal evaluation · Ultra · Without tools
DeepSeek V4.1 Flash · 36.8% — Max (100)Max (100) · Full HLE, pass@1, without tools. The separate text-only subset result is 39.1%.
Publisher sourceOpus 5.5 · 64.4% — MaxMax · Full 2,500-question HLE, without tools.
Publisher sourceHumanity’s Last Exam · Text-only, without tools
This text-only subset is separate from the full 2,500-question HLE and tools-enabled evaluations. No matching text-only Neurion Pro measurement was supplied.
GLM 5.3 · 42% — MaxIndependent evaluation by Artificial Analysis · Max · Text-only May 2025 revision; pass@1; no tools. The public comparison publishes a rounded 42%.
Artificial Analysis · Independent evaluationGPT-6.1 Sol · 53% — MaxIndependent evaluation by Artificial Analysis · Max · Text-only May 2025 revision; pass@1; no tools. The public comparison publishes a rounded 53%.
Artificial Analysis · Independent evaluationReturn to resultHumanity’s Last Exam · With tools
Tool access, context management and evaluation harnesses differ between publishers.
Neurion Pro · 58% — UltraHydraLabs-reported internal evaluation · Ultra · With tools
GLM 5.3 · 62.5% — MaxMax
Publisher sourceDeepSeek V4.1 Flash · 63.9% — Max (100)Max (100)
Publisher sourceGPT-6 Astra · 57.2%Publisher-reported evaluation
Publisher sourceOpus 5.5 · 67.7% — MaxMax
Publisher source - SciCodeReturn to result
Artificial Analysis currently marks SciCode under review following an independent dataset audit. The published integer percentages are retained without inventing decimal precision.
Neurion Pro · 69.6% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode
GLM 5.3 · 59% — MaxIndependent evaluation by Artificial Analysis · Max · Test-set subproblem scoring with scientific background; pass@1 over three repeats. The public comparison publishes a rounded 59%.
Artificial Analysis · Independent evaluationGPT-6.1 Sol · 54% — MaxIndependent evaluation by Artificial Analysis · Max · Test-set subproblem scoring with scientific background; pass@1 over three repeats. The public comparison publishes a rounded 54%.
Artificial Analysis · Independent evaluation - GPQA DiamondReturn to result
GLM and Sol use Epoch AI evaluations; DeepSeek and Astra retain publisher evaluations. Identical settings with the Neurion Pro measurement have not been established.
Neurion Pro · 94.8% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode · Epoch scope supplied by HydraLabs
GLM 5.3 · 90.91% — MaxIndependent evaluation by Epoch AI · Max · Epoch gpqa_diamond.csv, glm-5.3_max; mean score converted to a percentage and rounded to two decimals.
Epoch AI · Independent evaluationDeepSeek V4.1 Flash · 90.9% — Max (100)Max (100)
Publisher sourceGPT-6.1 Sol · 95.39% — MaxIndependent evaluation by Epoch AI · Max · Epoch gpqa_diamond.csv, gpt-6.1-sol_max; mean score converted to a percentage and rounded to two decimals.
Epoch AI · Independent evaluationGPT-6 Astra · 96%Publisher-reported evaluation
Publisher source - Terminal-Bench Science 0.1Return to result
Publisher scores are for version 0.1. Astra uses the later official 68.1% evaluation; its launch score was 64.6%. The Neurion Pro benchmark revision was not supplied.
Neurion Pro · 22.7% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode · Benchmark revision not supplied
GPT-6.1 Sol · 57% — MaxMax
Publisher sourceGPT-6 Astra · 68.1% — MaxMax · Later GPT-6.1 Sol release evaluation: 68.1%. The Astra launch result was 64.6%.
Publisher sourceOpus 5.5 · 58.7% — MaxMax · Safeguards enabled; fallback answered 3.9% of requests, affecting 5% of trials; 700 trials.
Publisher source - LiveCodeBenchReturn to result
Vals AI archived this benchmark on September 1, 2026 and no longer evaluates new model releases. No GPT-6.1 Sol result is listed; older Sol versions are not substituted.
Neurion Pro · 94% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode
GLM 5.3 · 80.53%Independent evaluation by Vals AI · Archived Vals AI LiveCodeBench evaluation, updated September 1, 2026; model GLM 5.3, not GLM 5.3 Flash.
Vals AI · Independent evaluation - CyberGymReturn to resultNeurion Pro · 94.9% — Ultra
HydraLabs-reported internal evaluation · Ultra · MelissaCode
GLM 5.3 · 84.5% — MaxMax · Claude Code 2.1.207, no web tools, pass@1 on 1,507 tasks.
Publisher sourceDeepSeek V4.1 Flash · 88.1% — Max (100)Max (100)
Publisher source - SEC-Bench ProReturn to resultNeurion Pro · 68.5% — Ultra
HydraLabs-reported internal evaluation · Ultra · MelissaCode · V8 and SpiderMonkey
DeepSeek V4.1 Flash · 62.8% — Max (100)Max (100) · Claude Code harness; pass@1.
Publisher sourceGPT-6.1 Sol · 78.8%Publisher pass@1 evaluation.
Publisher sourceGPT-6 Astra · 85.4%Publisher-reported evaluation
Publisher source - ExploitGymReturn to result
GLM and Opus task counts are converted to percentages and rounded to two decimals. Their six-hour budgets and the OpenAI evaluation settings differ.
Neurion Pro · 28.4% — Ultra — 6 hHydraLabs-reported internal evaluation · Ultra · MelissaCode · 6 hour budget
GLM 5.3 · 14.96% — Max — 6 hMax · 6 hour budget · 130/869 tasks; 6 h time-normalized budget including non-API overhead.
Publisher sourceDeepSeek V4.1 Flash · 15.3% — Max (100)Max (100)
Publisher sourceGPT-6.1 Sol · 35.1%Intended-vulnerability success rate per attempt.
Publisher sourceGPT-6 Astra · 42.4%Publisher evaluation without the 6 h time limit.
Publisher sourceOpus 5.5 · 34.52% — 6 h6 hour budget · 300/869 tasks at 6 h; cyber mitigations disabled.
Publisher source - ExploitBench · Coverage scoreReturn to result
These are capability-coverage scores, not full exploit-success percentages.
Neurion Pro · 74.2% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode · Coverage score
GLM 5.3 · 54.4% — MaxMax · Average coverage over 41 tasks; union across three revisions.
Publisher sourceGPT-6.1 Sol · 99.7% — MaxMax
Publisher sourceGPT-6 Astra · 100%Publisher-reported evaluation
Publisher sourceOpus 5.5 · 91%Cap% = 91% in plain and AutoNudge arms, not the separate 73.4% full arbitrary-code-execution rate.
Publisher source - SWE-bench VerifiedReturn to result
Vals AI archived this benchmark on September 1, 2026, before GPT-6.1 Sol’s release, and no Sol 6.1 result is listed. SWE-bench Pro, Multilingual, Multimodal and earlier model versions are not substituted for SWE-bench Verified.
Neurion Pro · 95.1% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode
GLM 5.3 · 95.4%Independent evaluation by Vals AI · Mini-SWE-agent · Independent Vals AI evaluation; 500 verified tasks; leaderboard updated September 1, 2026.
Vals AI · Independent evaluation - GDPval-AA v2.1Return to result
All competitor values are Artificial Analysis Elo scores from the same benchmark revision, checked October 12, 2026. Elo is not a percentage or a solved-task count. The Opus configuration includes default fallback. No Neurion Pro measurement has been supplied for this benchmark.
GLM 5.3 · 1651 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationDeepSeek V4.1 Flash · 1600 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationGPT-6.1 Sol · 1592 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationGPT-6 Astra · 1574 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationOpus 5.5 · 1866 Elo — Max · Default fallbackIndependent evaluation by Artificial Analysis · Max · Artificial Analysis Max configuration with default fallback. This is a system result that can involve fallback models.
Artificial Analysis · Independent evaluation - AA-Briefcase v1.1Return to result
All competitor values are Artificial Analysis Elo scores from the same benchmark revision, checked October 12, 2026. Elo is not a percentage or a solved-task count. The Opus configuration includes default fallback. No Neurion Pro measurement has been supplied for this benchmark.
GLM 5.3 · 1509 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationDeepSeek V4.1 Flash · 1422 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationGPT-6.1 Sol · 1557 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationGPT-6 Astra · 1570 Elo — MaxIndependent evaluation by Artificial Analysis · Max
Artificial Analysis · Independent evaluationOpus 5.5 · 1807 Elo — Max · Default fallbackIndependent evaluation by Artificial Analysis · Max · Artificial Analysis Max configuration with default fallback. This is a system result that can involve fallback models.
Artificial Analysis · Independent evaluation - Toolathlon VerifiedReturn to result
GLM 5.3 reports 73% on Toolathlon Verified. Opus 5.5 reports 77.8% pass@1 on Toolathlon with an internal harness; that result is not substituted for this Verified configuration. No matching GPT-6.1 Sol, Astra or DeepSeek V4.1 Flash result was verified.
GLM 5.3 · 73%Official evaluation service; pass@1 averaged over three independent runs.
Publisher source - ProgramBenchReturn to result
GLM and DeepSeek report the almost-solved metric. Opus 5.5 instead reports a 91.2% hidden-test pass rate on a 166-task subset, without the six-hour limit; this different metric is not merged into the chart. No matching GPT-6.1 Sol or Astra result was verified.
GLM 5.3 · 19%Publisher-reported Almost Solved metric: 19%.
Publisher sourceDeepSeek V4.1 Flash · 20.3% — Max (100)Max (100) · Almost@1: 20.3%; DeepSeek Harness Minimal; 1M context.
Publisher source - ALE-CLIReturn to result
This chart is the ALE-CLI configuration. DeepSeek V4.1 Flash publishes 31.8% for Agent’s Last Exam without identifying the CLI task set in its model card; it is not silently mapped to ALE-CLI. Older Sol and Opus versions are not substituted for GPT-6.1 Sol or Opus 5.5.
GLM 5.3 · 28.5% — MaxMax · Claude Code · Official ALE protocol; 105 isolated tasks; 1M context and 64K maximum output; default four-hour timeout with task-specific limits up to eight hours.
Publisher source - AutomationBench v1.0.6Return to result
Opus reports an early-access private held-out evaluation; GLM reports public v1.0.6. These are different evaluation sets.
Neurion Pro · 45.1% — UltraHydraLabs-reported internal evaluation · Ultra · MelissaCode
GLM 5.3 · 48.2% — MaxMax · Public v1.0.6, including the null-type handling fix.
Publisher sourceDeepSeek V4.1 Flash · 54.8% — Max (100)Max (100) · Official scaffold.
Publisher sourceGPT-6.1 Sol · 36.1% — MaxMax · Selected Max result: 36.1%. The published Medium result is 31.7%.
Publisher sourceGPT-6 Astra · 41.4% — MaxMax
Publisher sourceOpus 5.5 · 40% — MaxMax · Early-access private held-out evaluation; no fallback; interventions counted as failures.
Publisher source