🔍 Read the full analysis: Harness Adapters Offer A New Way To Balance AI Performance And Cost on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Hugging Face says reinforcement-learning LoRA adapters tuned for Pi, OpenCode and Hermes Agent outperformed the base Qwen3.8-27B model on two benchmarks. In the reported Pi test, pass@1 increased from 39% to 55%, while average token use fell from 2.9K to 1.5K per task; the supplied account does not include detailed scores for the other harnesses or independent replication.
Hugging Face says, in the original analysis, it used custom reinforcement-learning LoRA adapters to improve Qwen3.8-27B’s results in three AI agent harnesses: Pi, OpenCode and Hermes Agent. In its detailed Pi evaluation on Terminal-Bench 2.1, the team reports pass@1 rising from 39% to 55% and average usage falling from 2.9K to 1.5K tokens per task—a result that may matter to developers balancing agent performance with inference costs, but has not been independently replicated in the supplied account.
The project trained a separate adapter for each harness, the software environment that manages an agent’s interactions with tools and tasks. Hugging Face says each adapter beat the unmodified base model on two benchmarks under its matching harness. The supplied material gives a numerical comparison only for Pi, however, and does not identify both benchmarks in its summary or provide the OpenCode and Hermes Agent scores.
For Pi on Terminal-Bench 2.1, the team reports that the base model achieved 39% pass@1 and used an average of 2.9K tokens per task. With the model weights frozen and a custom LoRA adapter added, it reports 55% pass@1 and 1.5K tokens per task. Pass@1 measures whether a model completes a task successfully on its first attempt. The team attributes the change largely to less reasoning, fewer tool calls and more first-attempt successes.
Hugging Face says training and agent rollouts used its Hub infrastructure, including Jobs, TRL’s AsyncGRPOTrainer, OpenEnv and Harbor. The reported final training images used Pi 1.5.2, OpenCode 2.2.7 and Hermes Agent v0.22.6. According to the project, the harnesses ran in their shipped form, while a proxy recorded tokens; the team did not substitute a reimplementation of each agent loop.
Harness adapters tune models to the way agents work
Hugging Face reports that harness-specific LoRA adapters improved Qwen3.8-27B coding results while reducing token use. The detailed comparison provided is for Pi.
One adapter for each workflow
The team trained adapters inside the harnesses where agents use tools and complete tasks.
Detailed result reported
On Terminal-Bench 2.1, pass@1 rose from 39% to 55% with the adapter.
Harness-specific tuning
The project says its adapter beat the base model, but supplied figures do not quantify the result.
Same reported approach
A separate adapter was trained for this harness; detailed scores were not provided.
What changed in the Pi test
The team attributes the gains to less reasoning, fewer tool calls and more successful first attempts.
- 39% pass@1
- 2.9K average tokens per task
- 55% pass@1
- 1.5K average tokens per task
A recipe others can inspect
The project describes training in the shipped agent environments and says artifacts are available on Hugging Face Hub.
Tasks came from existing sources and generated examples, with validation checks described by the team.
The model weights stayed frozen in the reported Pi comparison while a custom LoRA adapter was added.
Adapters, GGUF files, tasks and a SafeAgents plugin are reported as available for inspection.
What the evidence leaves open
The reported gains are tied to the project’s experiments and evaluation setup.
- Numerical comparisons for OpenCode and Hermes Agent are not included.
- The supplied account does not provide uncertainty ranges or independent replication.
- More results are needed to assess the merged “super adapter” and performance on other task sets.
Why Harness-Specific Tuning Matters
Agent performance can depend not only on a model’s general capabilities but also on how it works through a particular harness. The harness shapes how the model uses tools and carries out a task. Training an adapter in that environment could, if the reported results hold up, improve task completion without requiring changes to the base model’s weights.
The reported token reduction is relevant to teams that run agents repeatedly because token use is one component of inference cost. Fewer tokens and tool calls could mean less computation per task, while a higher first-attempt success rate could reduce repeated work. But the figures are project-reported measurements, not a general cost guarantee: the supplied account does not translate token use into money or establish that the same gains would appear across other models, tasks or deployments.
The team also says it merged its three harness-specific adapters into one “super adapter” intended to run unchanged across Pi, OpenCode and Hermes Agent. A single adapter could simplify distribution if it performs reliably across those environments. The supplied material does not explain the merge method or give comparative measurements, so its performance cannot be evaluated from the information provided.
Products Worth Considering

Amazon Kindle Scribe (16GB) with Premium Pen, Tungsten

Jackery HomePower 3000 Portable Power Station, 3072Wh

Bose QuietComfort Wireless Bluetooth Noise-Cancelling Headphones, Cypress Green
As an affiliate, we earn on qualifying purchases.
Training Across Three Agent Environments
The project targets software-engineering tasks with a 27-billion-parameter model and extends earlier Hugging Face experiments in multi-harness reinforcement learning and harness adapters, which the supplied account says covered data-analysis and code-building work. The stated aim included producing a reusable training recipe, not just reporting scores.
For training, the team assembled coding tasks in Harbor format from existing sources and newly generated examples. These included SWE-smith tasks that introduce bugs into Python and Go repositories, SWE-Gym tasks packaged in TaskTrove and based on GitHub issues and merged fixes, and tasks created with Tasksmith from merged pull requests. The account says 24 tasks from 30 pull requests passed checks across 10 Python repositories. It also says OpenEnv validation and checks of task difficulty against the base model were used.
Task selection and verification can affect reinforcement-learning results because the model is trained against tasks and checks that reward solutions judged correct. The supplied account describes checks intended to identify faulty tests, but does not provide the full dataset composition or validation results. These details matter when other researchers try to reproduce the reported performance.
“Most of this is down to the model reasoning less, calling less tools, and performing better first time.”
— Hugging Face project team
Limits of the Reported Evidence
The supplied account includes detailed figures for only the Pi evaluation. It does not give the corresponding OpenCode or Hermes Agent scores, identify both benchmarks in its summary, or report sample sizes, uncertainty ranges and a full breakdown of task-level results. It is also unclear how the merged adapter compares with the three individual adapters.
Independent replication is not reported, and the material does not provide an independent audit of the task checks or enough information to assess whether the gains generalize beyond the selected evaluations. It also gives no direct cost calculation. The reported figures should therefore be understood as results from the project team’s experiments, rather than proof that every model or agent workflow will see similar gains.
Artifacts Open a Path to Replication
Hugging Face says the adapters, GGUF files, reinforcement-learning tasks and SafeAgents plugin are available through a collection on its Hub. Their availability gives developers and researchers a way to inspect the artifacts and try the recipe, although the supplied account does not state whether independent groups have done so.
The next useful evidence would include the complete benchmark results for all three harnesses, fuller details on task construction and evaluation, and comparisons of the merged adapter with the separate versions. Independent runs across additional tasks and deployments would help establish whether the reported performance and token-efficiency changes persist beyond the team’s tests. No further evaluation schedule is specified in the supplied material.
Key Questions
What did Hugging Face report?
Hugging Face says custom reinforcement-learning LoRA adapters improved Qwen3.8-27B’s results under Pi, OpenCode and Hermes Agent. The supplied account provides detailed numbers for Pi only.
How did the Pi result change?
On Terminal-Bench 2.1, the project reports that pass@1 rose from 39% to 55%, while average token use declined from 2.9K to 1.5K per task. These are results from the team’s experiment.
Does the report show lower dollar costs?
No direct cost figure is provided. Lower token use could affect inference costs, but the supplied material does not calculate savings in money or establish how costs would change in a particular deployment.
Have the findings been independently verified?
The supplied account does not report independent replication. It says the project’s adapters, tasks and related files are available on Hugging Face Hub, allowing others to inspect or test them.
What information is still missing?
The account does not provide detailed scores for OpenCode or Hermes Agent, full benchmark and task-level results, uncertainty ranges, or a comparison of the merged adapter with the separate adapters.
Primary source: Hugging Face · via ThorstenMeyerAI.com
