🔍 Read the full analysis: Same AI Model, Different Results: Understanding CUGA, Hermes, And Deep Agents on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
IBM reports that CUGA completed 73 of 90 AppWorld tasks using GLM-5.3-flash, compared with 42 for Hermes and 27 for Deep Agents. The results show a difference among these setups on one benchmark, not a general ranking of the agent systems or evidence about policy enforcement.
IBM reports in the original analysis that its CUGA agent harness completed 73 of 90 AppWorld tasks using the same open-weight model as two alternatives, compared with 42 for Hermes and 27 for Deep Agents. The comparison matters because it suggests that an agent’s software setup can affect task completion even when the underlying model is held constant, but the result covers one benchmark and does not establish an overall ranking.
The three systems ran GLM-5.3-flash on the same 90 tasks across 30 scenarios, IBM says. Runs used the Harbor evaluation framework and one attempt per task. IBM reports pass rates of 81.1% for CUGA, 46.7% for Hermes and 30.0% for Deep Agents. On a separate scenario measure, a scenario counted only when all three of its variants passed: CUGA passed 20 of 30 scenarios, Hermes eight and Deep Agents two.
The harness configurations were not identical. CUGA used task decomposition, app-focused tools, verification before execution and automatic reflection. Hermes ran with Harbor’s default setup and AppWorld tools over MCP. Deep Agents used AppWorld tools over MCP with LangChain middleware, including tool selection, context summarization, a to-do list, retries and call limits. IBM also says it made two fixes to Deep Agents so malformed model replies would not end runs.
IBM says the experiment setup is publicly available in harbor-eval, its Harbor fork. AppWorld tests multi-step tasks across nine apps and 457 APIs, including email, payments and file storage. IBM describes the benchmark as a proxy for some capabilities involved in enterprise automation, such as reading API documentation, paging through results and changing state across apps. It uses consumer apps, however, and does not directly establish performance on real enterprise systems.
Why Harness Design Affects Results
The reported gap—from 27 to 73 completed tasks with the model held constant—shows why a model score alone may not predict how an agent performs. The harness determines how work is divided, which tools are available, and whether actions or outputs are checked. Those implementation choices can shape whether a multi-step task succeeds.
For organizations evaluating agents, the practical takeaway is to test the specific combination of model, harness and workload they expect to use. The AppWorld results are a signal for workflows resembling its API-based tasks, not proof that CUGA will do better in personal assistance, research or other settings for which Hermes and Deep Agents are designed.
The scores also do not test whether an agent follows organizational rules during execution. IBM describes CUGA as supporting rules through its harness, but these benchmark results do not establish policy enforcement. Access controls, approval steps, data boundaries, cost and latency would require separate evaluation.
Products Worth Considering
Amazon Kindle Scribe Colorsoft, 64GB, 11-Inch Color Display with Premium Pen
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Bose SoundLink Plus Portable Bluetooth Speaker, Wireless Outdoor Speaker, Up to 20 Hours Battery Life, Waterproof and Dustproof, Black
As an affiliate, we earn on qualifying purchases.
Three Harnesses, Different Use Cases
The comparison brings together systems presented for different kinds of work. The source describes CUGA, short for ConfigUrable Generalist Agent, as IBM’s open-source harness for enterprise tasks across systems such as customer relationship management and payments. It describes Hermes from Nous Research as designed for always-on personal assistants and Deep Agents from LangChain as suited to long, multi-step work such as research.
That difference in intended use is relevant: the AppWorld test is a comparison on a defined workload, not a contest that settles which product is best overall. The source also cites directories that track nearly 200 agent harnesses and a separate reported example in which GLM-5.2’s SWE-bench Pro result rose from 23% to 52% when the harness changed. That is a different comparison and is not evidence from the AppWorld experiment.
“This is not a verdict on Hermes or Deep Agents.”
— IBM, describing the AppWorld results
Limits of the AppWorld Comparison
The results cover one model, one 90-task sample and one benchmark, with a single attempt per task. IBM says the harness setups differed, so the experiment does not isolate every implementation difference. The supplied material does not show how results would change with other models, more tasks, repeated runs or production workloads.
It is also unclear whether the reported advantage would hold on systems used by actual enterprises. AppWorld uses consumer apps, and the experiment did not evaluate policies. The source provides no comparative figures for cost, latency, large-scale reliability or deployment outcomes, and it gives no independent replication of the results.
Evaluate on Representative Workflows
The experiment setup is available through IBM’s harbor-eval fork, according to the source. Teams considering these systems can use it as a reference while testing the model and tool configurations they expect to deploy against workflows representative of their own operations.
That testing should track successful completion across varied and repeated tasks, as well as requirements the AppWorld comparison did not measure, including access controls, approval steps and data boundaries. The supplied material gives no timetable for a broader comparison or independent replication, so whether the reported differences persist beyond this test remains open.
Key Questions
What did the AppWorld test find?
IBM reports that CUGA completed 73 of 90 tasks, compared with 42 for Hermes and 27 for Deep Agents, with all three using GLM-5.3-flash.
Does this prove CUGA is the best agent?
No. The result applies to one benchmark, one model and one task sample. It does not establish a general ranking across other workloads or systems.
What is an agent harness?
A harness is the software and configuration that coordinates an agent’s work, including task planning, tool access and checks around actions or results.
Did the experiment test enterprise policy enforcement?
No. The reported experiment did not use policies, so its scores do not show whether any system reliably enforces organizational rules.
What should organizations test next?
Organizations should compare the intended model-harness combination on representative workflows, measuring repeated-task consistency and requirements such as access controls, approvals and data boundaries.
Primary source: Hugging Face · via ThorstenMeyerAI.com
