Performance Comparison of CUGA, Hermes, and Deep Agents
The CUGA team conducted a controlled experiment using the same open-weight model, GLM-5.3-flash, across three different agent harnesses on 90 AppWorld tasks. The results revealed significant differences in performance: CUGA successfully completed 73 out of 90 tasks (81.1%), while Hermes completed 42 tasks (46.7%) and Deep Agents completed 27 tasks (30.0%). This experiment utilized the Harbor evaluation framework, with each task attempted once.
Understanding AppWorld and the Experiment Setup
AppWorld serves as a benchmark for multi-step tasks across nine applications and 457 APIs, designed to simulate multi-app automation scenarios such as logging in, paging, API calls, and state changes. Each run employed the same model (GLM-5.3-flash) and the same set of 90 tasks distributed across 30 scenarios. However, the configurations of the harnesses varied significantly:
- CUGA: Included AppWorld instructions, task decomposition, verification before execution, and automatic reflection.
- Hermes: Utilized the default setup with AppWorld tools over the MCP framework.
- Deep Agents: Incorporated AppWorld tools over MCP, along with LangChain middleware for tool selection, context summarization, retries, and limits.
Results and Their Interpretation
The report presents two key metrics: TGC (tasks completed) and SGC (scenarios where all three harnesses succeeded). The findings were as follows:
- CUGA: 81.1% (73/90) TGC and 66.7% (20/30) SGC
- Hermes: 46.7% (42/90) TGC and 26.7% (8/30) SGC
- Deep Agents: 30.0% (27/90) TGC and 6.7% (2/30) SGC
It is important to note that these results do not serve as a definitive judgment on Hermes or Deep Agents. Instead, they highlight the suitability of each harness for specific types of tasks, particularly those involving multi-app API interactions.
Limitations of the Study
The authors acknowledge several limitations in their analysis:
- The 90-task sample may not be representative of other task types.
- The setups for each harness were not identical, which could influence the results.
- The runs did not incorporate policy enforcement, meaning the experiment does not measure the impact of policy adherence on performance.
CUGA’s Unique Features and Policy Enforcement
CUGA’s superior performance is attributed to its unique features that minimize decision points for the model and introduce runtime checks. Key features include:
- Task decomposition and focused tools
- A tool shortlister for identifying necessary tools
- CodeAct, which generates Python code executed in a sandbox environment
- Verification before execution to ensure code integrity
- Automatic reflection after actions to review outcomes
- Pagination audits to manage data retrieval
Each step controlled by the harness serves as an enforcement point for business rules. The team also discusses a policy layer described in their paper “Governance by Construction for Generalist Agents” (ACM CAIS 2026, arXiv), which can enforce rules at four critical points during execution:
- Intent Guards: Block requests that match certain criteria.
- Playbooks: Provide standard procedures for agent guidance.
- Tool Approval: Require human approval before executing certain actions.
- ToolGuard: Implement checks that can block calls based on predefined rules.
Although the AppWorld runs did not utilize these policies, the report illustrates their potential impact through a Policy Playground example, demonstrating how policy enforcement can alter outcomes.
Deployment and Future Testing
CUGA is licensed under Apache 2.0, supports OpenAI-compatible endpoints, and can be deployed locally or on Kubernetes within private or sovereign cloud environments, including air-gapped clusters. It also features budget controls and detailed run receipts for transparency. The CUGA team has made the comparison setup and Harbor fork (harbor-eval) publicly available and is open to rerunning the benchmark if alternative configurations for Hermes or Deep Agents are submitted for the same tasks.
Original source: Hugging Face Blog