Signal SentrySearchUpdated Nov 20, 05:30 UTCAM Drop
Frontier watch · Orchestration & memory

Agent orchestration: does the harness beat the model?

Does the orchestration layer (decomposition, dispatch, verification, routing between agents) matter more than the base model, and which designs work?

Why it matters

If orchestration rather than the model decides whether long agent jobs finish, builders get more from how they split, check and route work than from switching models, and costs depend on how many agents a design spins up.

State of play

Newest first. Each claim is our summary of the linked primary source, with a short quote from it.

  1. ResearchOct 15, 2026Source

    A preprint reports that HTN-Orchestrate, which defers task decomposition until prerequisite results arrive, beat ReAct and Agent Team with the same base model: 89.73% on BrowseComp and 73.44% on SUPERChem.

    HTN-Orchestrate outperforms ReAct, Agent Team, and DeepSeek Harness using the same base model, achieving 89.73% accuracy on BrowseComp and 73.44% on SUPERChem, exceeding Agent Team by 2.45 and 4.69 percentage points, respectively.

    What it does not show: Preprint; margins over Agent Team are 2.45 and 4.69 points; results are the authors' own.

  2. ResearchOct 13, 2026Source

    Sakana AI researchers report in a preprint on The Agentic Scientist that their ablations suggest reliable scientific autonomy depends on the cross-agent orchestration layer as well as base model capability.

    Our ablations suggest that reliable scientific autonomy depends not only on base model capabilities, but also on the cross-agent orchestration layer.

    What it does not show: Preprint; Sakana's project page also says the choice of coding harness (Codex, Claude Code, OpenCode, Pi) had a smaller effect than the underlying model.

  3. DocumentedOct 9, 2026Source

    Anthropic's release notes say Claude Managed Agents agents can now write dynamic workflows, in beta: a program that runs many agents in phases and combines their results, run by the server in the background.

    A workflow is a program that runs many agents in phases and combines their results.

    What it does not show: Beta. Anthropic's docs cap a run at 1,000 agents over its life and 64 working at once, so not 1,000 in parallel; no outcome data here.

  4. ResearchOct 9, 2026Source

    Researchers report in a preprint that sizing multi-agent collaboration to the backbone model's competence raised average accuracy from 87.22 to 89.97 for stronger models while cutting token cost by 36.26%.

    for higher-competence backbones, it improves average accuracy from 87.22 to 89.97 while reducing token cost by 36.26%; for lower-competence backbones, it yields a further average gain of 4.58 points over existing methods.

    What it does not show: Preprint; six reasoning benchmarks, not long-running agent tasks; self-reported.

  5. ResearchOct 7, 2026Source

    A preprint comparing matched Codex, Claude Code and Kimi Code runs with and without dynamic sub-agent spawning finds model capability drives short tasks, while orchestration becomes central on long-horizon work.

    Model capability largely determines outcomes on shorter tasks, whereas long horizon development makes orchestration central to task completion.

    What it does not show: Preprint; 354 tasks and 2,124 executions; it catalogues 13 concurrency-specific failure modes rather than reporting one win rate.

  6. ResearchSep 25, 2026Source

    Researchers report in a preprint that on AgentWorld, 100 tasks needing 3-20 agents over 50+ rounds, the best model tested reached 52.0% task success, with failures including communication breakdowns and lost shared plans.

    Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds.

    What it does not show: Preprint; four smaller models were tested (Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, DeepSeek R1-70B), not the labs' flagships.

What would change the picture

Signals we are watching for. Our own reading of where the thread could go: not predictions, and not sourced claims.

  • An independent benchmark scores the same base models under several orchestration designs and finds a consistent gap across labs' models Would suggest: Orchestration is a separable lever, not an effect of one vendor's model or harness
  • A flagship model clears 80% task success on a long-horizon multi-agent collaboration benchmark such as AgentWorld Would suggest: Coordination across many rounds and agents is becoming reliable rather than the main failure point
  • Anthropic or OpenAI moves its API multi-agent workflow feature from beta to general availability Would suggest: Fan-out orchestration becoming standard platform infrastructure rather than an experiment
  • A study shows multi-agent runs beating a single agent at an equal token budget on long tasks Would suggest: The gains come from the design, not just from spending more compute

From the Drops

AgentsResearch

Preprint: multi-agent LLMs matched or trailed equal-budget baselines in 74% of tests

A synthesis of nine studies and 516 configurations finds multi-agent accuracy averaged -0.042 against baselines matched on samples or budget. The author argues gains need split evidence, outside checks or varied models.

Why it matters: Before adding agents, compare against one model given the same samples or budget; extra agents reading identical evidence may add cost without accuracy.

From Drop #095 · Thu, Nov 19, 2026 · Morning
AgentsShipping

Google's Sashiko code-review agent: 1200 Linux reviews a day, fewer follow-up fixes

Google says Sashiko, in production for seven months, splits bug-finding from verification across specialist agents. Kernel commits needing a fix within 60 days went from 3.4% to 2.2% by Linux 7.2; Google won't credit Sashiko alone.

Why it matters: Evidence that an agent reviewer can earn maintainers' trust when each finding and each dismissal must be proven in code. Try separating finder agents from verifier agents in your own review.

From Drop #094 · Wed, Nov 18, 2026 · Evening
AgentsResearch

Runta: 64.5% of coding-agent spend in 27,280 runs went to trials that failed

Runta and Baseten ran 22 harnesses with 5 open models on 62 tasks. Runta says the model explained 92.8% of cost variance, but with the model fixed the harness moved cost per task 1.8 to 5.0 times.

Why it matters: Switching models may call for switching harnesses too, Runta says. Measure cost per passed task on your own tasks, and add a check before the agent declares done; Runta did not test that fix.

From Drop #092 · Tue, Nov 17, 2026 · Evening
AgentsResearch

Preprint: a leaner harness lifted robot-arm agent scores from 29.7 to 50.3

In a preprint testing robot-use agent designs side by side, researchers say a lean harness keeping full episode history goes far and scene geometry helps most; their simple harness lifted RoboDojo scores from 29.7 to 50.3.

Why it matters: Before paying for a new model, test what your harness keeps and drops. The authors got their gain with no learned policy or task-specific tuning, though only in simulation.

From Drop #087 · Sun, Nov 15, 2026 · Morning
AgentsResearch

Preprint: auto-tuning a multi-agent harness lifted accuracy, left orchestration failures

Researchers report in a preprint that MULTILOOP, a loop where a coding agent edits a multi-agent harness, lifted DeepSearchQA accuracy 4.6% over the base harness. Orchestration failures largely persisted.

Why it matters: Automated harness tuning may patch symptoms without fixing how agents delegate and share information. Review the handoffs directly rather than trusting a higher score.

From Drop #085 · Sat, Nov 14, 2026 · Morning
AgentsResearch

Preprint: delegating to sub-agents gave no consistent accuracy gain in generative search

Researchers report that delegating to tools and other LLMs gave no consistent accuracy gain over non-delegated setups, and could cut accuracy sharply on post-cutoff questions, where the same LLM using tools directly did much better.

Why it matters: Adding sub-agents is not a free upgrade. Before splitting a search task across agents, compare it with one model calling the tools directly on questions past its cutoff.

From Drop #084 · Fri, Nov 13, 2026 · Evening
ModelsAnnounced

Google ships Gemini 4 Hydrogen at $1.50/$7.50, plus an Orchestrator that mixes models

Google DeepMind says Hydrogen, its fast Gemini 4 model, finishes tasks 2.5 times faster than 3.8 Flash and scores 57% on Terminal-Bench 4.1. Its Orchestrator in Antigravity switches between Carbon and Hydrogen in one session.

Why it matters: If you run Gemini 3.8 Flash or pay frontier prices for routine agent steps, test Hydrogen as the executor; Google says Orchestrator neared Carbon's success rate on 24% less quota.

From Drop #083 · Fri, Nov 13, 2026 · Morning
AgentsResearch

Preprint: how a model organizes its sub-agents measurably changes results

WorkflowArena tests nine models on 145 long-horizon tasks in 8 domains. The authors report no setup wins everywhere, and that weaker models do much better when Claude Opus 5.5 writes the workflow they execute.

Why it matters: If a model spawns and organizes your sub-agents, evaluate the planner apart from the workers; the authors treat orchestration as its own capability.

From Drop #078 · Tue, Nov 10, 2026 · Evening
AgentsResearch

Preprint: many small agent teams beat one big team at the same compute budget

Researchers report that with a 27B model and 100 agents, parallel teams of ten beat independent agents on 10 of 12 long-horizon tasks and one 100-agent team on all four tasks tested; at matched spend they topped a frontier model on half.

Why it matters: Before growing one agent team larger, try several small teams and keep the best result. It depends on having a verifier that can score each attempt.

From Drop #076 · Mon, Nov 9, 2026 · Evening
AgentsResearch

Preprint: reusable agent primitives lift Claude Code and Codex from 9 to 39 and 35 tasks

Researchers report that on 46 real-world tasks, Claude Code running Opus 5.5 and Codex running GPT-6.1 Sol each finished 9 alone, then 39 and 35 once wired to their HEELO MCP server, at 1.55 times lower cost per completed task.

Why it matters: If your agents stall when applying a tool or resource, the authors place the fix in the harness, not the model. Treat it as a preprint on the authors' own platform until others reproduce it.

From Drop #075 · Mon, Nov 9, 2026 · Morning
AgentsDocumented

LangChain adds async subagents so an agent stays reachable while work runs

LangChain says Deep Agents can now start subagents in the background, then check, update, cancel or list them, keeping the orchestrator free for the user. LangChain labels the task alerts beta and subject to change.

Why it matters: Long tasks no longer have to block the agent your users talk to. If you build agents, test how approvals and mid-run corrections behave when several background tasks run at once.

From Drop #069 · Fri, Nov 6, 2026 · Morning
AgentsResearch

Google benchmark: frontier agents score under 50% on real engineering projects

Google's PLE, built with Mercor, has 85 long-horizon tasks in four engineering fields. Across 14 models and seven harnesses, the best mean score was 48.3%; only 41.3% of the top setup's deliverables passed every hard safety check.

Why it matters: Per Google, the model moved scores far more than the harness did. For engineering work, pick the model first and keep a licensed engineer signing off on every design.

From Drop #065 · Wed, Nov 4, 2026 · Morning
AgentsDocumented

LangChain: browser toolsets passed 78% to 90% of tasks, but cost varied about 13x

Holding the model fixed on WebVoyager, LangChain says Stagehand passed 90% for $1.98 total while Playwright MCP passed 78% for $26.71. Ending fixed sleeps once a page went quiet cut median HardBench time from 182 s to 124 s.

Why it matters: Tool design, not only the model, sets an agent's cost and speed. Trace where your agent burns turns and tokens, and test each tool change with evals before keeping it.

From Drop #065 · Wed, Nov 4, 2026 · Morning
AgentsResearch

Preprint: goal-stack recovery lifts AppWorld success up to 22.07 points over ReAct

Researchers report the Verify-Backtrack Loop, a training-free ReAct extension, keeps subgoals in a bounded stack and checks feasibility on failure. It beat ReAct and reflection on every model and benchmark pair they tested.

Why it matters: Long agent runs fail at recovery, not length, the authors argue. If your agent retries blindly after errors, goal tracking in the harness is a cheap, training-free change to test.

From Drop #062 · Mon, Nov 2, 2026 · Evening
AgentsResearch

Preprint: coding model's harness edits lift frozen robot policies from 9.7% to 43.3%

Researchers report RoboArchitect has a coding model watch rollouts, diagnose failures and rewrite the harness, with no weight updates: average success on unseen scenes rose from 9.7% to 43.3%, on unseen tasks from 12.7% to 40.2%.

Why it matters: Before fine-tuning for a new setup, test whether a better harness closes the gap. The authors suggest much of the headroom sits in the harness; this is a preprint, not a peer-reviewed result.

From Drop #061 · Mon, Nov 2, 2026 · Morning
AgentsResearch

Preprint: a planner-coder-reviewer pipeline cost 3.22x a single call, solved no more

Across 135 HumanEval and ClassEval tasks in 2430 runs with Claude Opus 5.5 and Haiku 5.5, researchers report protocol handoffs added only 7.2 ms each, but decomposition did not pay: 3.22x the cost, 3.24x the time.

Why it matters: The authors report the saving came from picking a model per task: an oracle choice between the two single calls solved 94.3% of runs at half the strong call's cost per solved task.

From Drop #056 · Fri, Oct 30, 2026 · Evening
AgentsResearch

Preprint: LLM research agents built a Hearthstone Battlegrounds AI with 80% first places

Researchers report AutoTavern's agents, with a human setting goals, built a game simulator and 23 versions of TavernAgent in 20 days. Its last 20 live games averaged a 1.60 placement with 80% first places.

Why it matters: The authors find LLM agents build far better than they play: TavernAgent answered 91% of play tasks, against at most 63% for frontier LLMs.

From Drop #053 · Thu, Oct 29, 2026 · Morning
AgentsResearch

Preprint: well-timed handoffs to cheaper models cut agent API costs 65.9% on average

Researchers report that switching from a capable to an efficient model at the right stage largely kept accuracy, while the reverse did not. Their AgentBaton detector averaged 65.9% savings across 11 model pairs.

Why it matters: A simple cost lever for agent builders: one well-timed handoff rather than routing every step, per the authors' two benchmarks.

From Drop #052 · Wed, Oct 28, 2026 · Evening
AgentsResearch

Preprint: at Volvo, off-the-shelf agent harnesses cut monthly test costs 3-66%

Researchers compared Volvo Group's in-house GUI-testing harness with Claude Code and OpenCode, keeping MCP tools and models the same. On infotainment tests success matched, but inference cost per task was 2.6-10.7 times higher.

Why it matters: Counting engineering effort, a nightly regression suite cost 3-66% less a month, the authors report. Testers still wanted to inspect the agent's actions.

From Drop #050 · Tue, Oct 27, 2026 · Evening
AgentsResearch

Preprint: DynaFlow trains a main agent to write its own multi-agent workflows

The main agent decides when to call sub-agents and generates programs that run them in sequence or in parallel, with verification. Only the main agent is trained; the sub-agents stay fixed.

Why it matters: With the same Qwen3.5-27B backbone, the authors report 88.08% overall accuracy across six benchmarks, 10.90 points above the strongest baseline they tested.

From Drop #050 · Tue, Oct 27, 2026 · Evening
AgentsResearch

Preprint: the same coding model's success swung 44 points across harnesses

Researchers paired six open-source harnesses with three lightweight models on 100 SWE-bench Verified tasks, 1,800 graded runs. Harness rankings changed across models, the preprint reports.

Why it matters: Tokens per resolved task varied more than 20-fold even among similar harnesses, the authors say, making harness-model selection a central deployment decision.

From Drop #048 · Mon, Oct 26, 2026 · Evening
AgentsResearch

Preprint: realistic MCP tool faults cut agent task scores by 60%, and scale didn't help

Researchers built MCP-Injector, with faults calibrated from 57,240 threads across 1,774 MCP repos. Task-completion score fell from 8.08 to 3.23 out of 10; a 120B gpt-oss kept no more of its score than a 20B.

Why it matters: The authors report agent stacks differed in absorbing faults, suggesting framework error handling matters more than model scale. Capability benchmarks can hide this.

From Drop #047 · Mon, Oct 26, 2026 · Morning
AgentsResearch

Preprint: IF-MAP picks which agent in a multi-agent team to improve first

Researchers report that across six multi-agent workflows, tuning the prompts of the agents IF-MAP selects gained 3.81 percentage points on average, versus 1.85 when agents were picked uniformly.

Why it matters: The authors say it cut Qwen3-8B token use by 74.2% versus optimizing every agent: a cheaper way to aim improvement effort in agent teams.

From Drop #045 · Sun, Oct 25, 2026 · Morning
AgentsResearch

Preprint: a team of AI researchers with varied expertise beats GDN2 at 1.3B scale

Researchers report a team of persistent AI researchers from different disciplines, with private histories and shared evidence, found linear attention designs beating GDN2 at 1.3B scale; removing any piece weakened results.

Why it matters: The authors argue that who explores, what they remember and how effort is allocated drive autonomous discovery, not only stronger agents.

From Drop #044 · Sat, Oct 24, 2026 · Evening
AgentsResearch

Preprint: a shared memory buffer beats agent debate with 77.4% fewer tokens

Researchers report in a preprint that Muffin, where agents post findings to a shared buffer asynchronously, had the best accuracy in 17 of 20 settings and gained 19.2 points over multi-agent debate with 77.4% fewer tokens.

Why it matters: How agents share findings may matter more than how many agents you run; worth testing before adding more agents.

From Drop #043 · Sat, Oct 24, 2026 · Morning
AgentsDocumented

AWS shares how its Deep Insight multi-agent analyst made it to production

AWS's Solutions Architecture team says eight agents split the work: a Haiku model routes requests, an Opus planner drafts steps for user approval, and a Validator re-runs each calculation. A code interpreter runs on AWS Fargate.

Why it matters: The lessons: route easy requests to small models, plan before acting, verify numbers. Note the web version auto-approves a plan after 300 seconds with no reply.

From Drop #039 · Thu, Oct 22, 2026 · Morning
AgentsResearch

Preprint: multi-agent setups lost to single agents on τ²-bench's user-facing tasks

Across about 7,950 simulated conversations and six models, researchers report the multi-agent system was worse in 12 of 13 valid cells and tied in one, with a weighted mean gain of -28 points.

Why it matters: For agents that serve people, the authors say a single agent should be the default, and a multi-agent design must show compute-matched evidence from its own setting.

From Drop #038 · Wed, Oct 21, 2026 · Evening
AgentsResearch

Preprint: evolving harness and policy together lifts a coding agent from 57.8% to 64.8%

Researchers report Self-HPO alternates harness changes the model proposes from its own failures with RL on the model. Across three coding benchmarks it raised Qwen3.5-35B-A3B's average resolve rate.

Why it matters: The authors say the joint gain beat the sum of harness-only and policy-only gains: a data point for teams deciding whether to tune the harness or the model.

From Drop #035 · Tue, Oct 20, 2026 · Morning
AgentsResearch

Cohere: a strong lead with cheap agent workers neared frontier results at 70% lower cost

On 200 ProgramBench tasks, Cohere says a Sol lead with 16 cheap Luna planning workers hit 61.8% reward vs 63.2% for an all-Sol team, cutting cost from $510 to $155. Teams gained up to ~10 points over solo agents at 3–7× the cost.

Why it matters: Cohere found where you put agents matters more than how many: delegating only planning matched delegating every stage (63.6% vs 63.2%) at half the cost.

From Drop #034 · Mon, Oct 19, 2026 · Evening
AgentsResearch

Preprint: agent swarms that bid for turns learn to organize without a boss agent

Researchers report in a preprint that market-based post-training lets agents bid to decide who acts. Tested on crafting, robot navigation and deep research, learned self-orchestration beat other coordination strategies.

Why it matters: The authors report trained swarms adapted faster to related tasks and lost less performance when agents dropped out, a hint for decentralized agent designs.

From Drop #034 · Mon, Oct 19, 2026 · Evening
AgentsResearch

Preprint: MS-Eval finds sub-agent delegation often raises cost with uneven payoff

Testing four main-agent and sub-agent designs with eight frontier models on MCP-heavy tasks, researchers report specialized sub-agents work best. Delegation helped most open-source models but cost closed-source ones performance.

Why it matters: Gains came from context offloading and real parallelism, the authors say; unnecessary or redundant delegation turned the same cost into pure overhead.

From Drop #033 · Mon, Oct 19, 2026 · Morning