Signal SentrySearchUpdated Nov 20, 05:30 UTCAM Drop
Agents · by industry

Who put an agent to work.

Deployments, launches, research and field reports the Drops covered. Each shows its stage, whether a human is in the loop, and its sources.

From the Drops

AgentsConsumer softwareLiveHuman in loop: yes

Gemini can now operate your Mac as Apple readies limits on full-access agents

Google's Gemini Mac app can now act across local files, apps and the web, limited to apps the user picks, The Verge reports. Apple's macOS 27.3 beta previews tighter full disk access rules; Bloomberg says Apple is building one too.

Why it matters: Desktop agents can now reach most apps on a work machine. Decide which apps each agent may touch before staff switch this on, and watch for Apple's new access prompts.

From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsCloud softwareLiveHuman in loop: no

AWS lets agents pay Marketplace sellers over x402 on the AWS bill, under a session cap

AWS says AgentCore payments now settles agent purchases from x402 Marketplace sellers in USD on the AWS bill, with no stablecoin wallet. Per-session caps, signed quotes and Private Marketplace limit what an agent can spend and buy.

Why it matters: Agent spending can now sit inside existing cloud budgets and audit trails. In AWS's sample the agent can pay but cannot open sessions, so it cannot lift its own budget: copy that split.

On the Frontier watch →From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsData engineeringTestingHuman in loop: yes

Databricks: over half of 80,000-plus pipeline issues needed in-house knowledge to fix

Databricks says its ZeroOps pipeline-repair agent saw over 80,000 customer issues at hundreds of companies in preview. Now in public beta, it adds custom instructions and one-click pull requests, and a person reviews every fix first.

Why it matters: Generic agents miss house rules. Before judging any ops agent, budget time to write down your flaky sources, preferred fixes and naming conventions for it.

From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsAutomotiveLiveHuman in loop: yes

BMW sets agent autonomy per step, not per agent, across 12,500 R&D developers

BMW Group says each workflow step gets one of three trust levels, from human sign-off to autonomy within policy. Checks run in CI/CD outside the agent, and a central registry screens agents for prompt injection and duplicates.

Why it matters: Trust can grow one step at a time instead of all at once. Map which steps of your agent workflows need sign-off, and move policy checks out of prompts and into code.

From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsWeb browsingTestingHuman in loop: unclear

Google built a walled-off copy of the web to test Chrome agents against hidden attacks

Wired reports Chrome's team built Mirror Dimension, an isolated web of agent-built shopping, social and bank sites where browsing agents meet hidden prompt injections. Google says it shares the approach so other teams consider it.

Why it matters: Testing browser agents on the live web or in a toy sandbox misses real attacks. If you ship one, ask whether your tests include realistic sites carrying planted instructions.

From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsEnterprise softwareLiveHuman in loop: unclear

Preprint: a weaker agent did better with 29 of its learned guidelines than all 199

Researchers whose memory pipeline runs in IBM's CUGA agent report gpt-oss-120b went from 39.9% to 51.2% on AppWorld with all 199 guidelines, and 55.4% with about 29. A strong model kept gaining as guidelines grew.

Why it matters: More memory is not free accuracy. Size agent memory per model, and keep provenance when lessons are merged; the authors say theirs was silently dropped.

On the Frontier watch →From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsDeveloper toolsTestingHuman in loop: unclear

Gemini API previews Tool Retrieval, loading tool schemas only when an agent needs them

Google says agents can keep tool libraries, remote MCP servers included, on standby: the model sees names and descriptions and fetches full schemas on demand. No extra charge; works with Gemini 3.6 and later in the Interactions API.

Why it matters: Agents with dozens of tools pay for every schema on every call. Google says savings come when unused schemas outweigh loaded ones, so measure on your own traffic.

From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsSoftwareTestingHuman in loop: unclear

LangSmith Tool Gateway puts agent tool credentials, deny rules and approvals in one layer

In public beta, LangChain's gateway sends each MCP tool call through one URL, decides whether it runs as the user or the agent, applies deny and approval policies, and records who called what.

Why it matters: If your agents hold hand-rolled OAuth tokens or see a whole tool server, this is the pattern to copy or buy: scoped tool kits, identity per call and an audit trail.

On the Frontier watch →From Drop #096 · Thu, Nov 19, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Preprint: crashed agent tool calls repeated in 117 of 200 replays without a full runtime

A preprint injected ten fault types into an agent runtime. With a ledger, reconciliation and evidence sealing, 0 of 200 crash replays repeated an effect versus 117 for simpler setups, at about one-seventh the throughput.

Why it matters: Idempotency keys alone don't tell you whether an interrupted action happened; check that your agent runtime reconciles against the target system before it retries.

On the Frontier watch →From Drop #096 · Thu, Nov 19, 2026 · Evening
AgentsSoftwareLiveHuman in loop: yes

Snowflake's coding-agent rollout: name what works, then scale it, Stack Overflow reports

Stack Overflow says Snowflake SVP Vivek Raghunathan's team ran a five-stage rollout, gathered roughly 14 named practices such as one git worktree per agent, and spread them through org-wide focus weeks.

Why it matters: If adoption is high but results are uneven, measure breadth of use first and give someone senior the job of documenting what your best engineers do before redrawing roles.

From Drop #096 · Thu, Nov 19, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Preprint: trying a small open model first cut coding-agent GPU-hours up to 33%

Researchers tested 36 open-weight models and say small ones fall short mainly on hard tasks. Trying a small model first and escalating only when needed cut GPU-hour costs up to 33% on SWE-bench Verified, they report.

Why it matters: Before buying a learned router, test a plain small-first fallback on your own coding tasks; the authors say it nearly matched an oracle-style router.

From Drop #096 · Thu, Nov 19, 2026 · Evening
AgentsAutomotiveLiveHuman in loop: unclear

AUMOVIO routes all model access through one Databricks gateway with Entra ID sign-in

AUMOVIO says vendor-neutral model aliases let it re-point apps to new providers with no code changes, apps get access by exchanging an Entra ID token instead of API keys, and spend is charged to cost centers.

Why it matters: When access ends with the corporate identity, offboarding covers AI too; check whether your agents still run on shared API keys that outlive the people who created them.

From Drop #096 · Thu, Nov 19, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Preprint: coding agents kept working after a correct patch, and 1–8% then undid it

Across five models, two frameworks and three SWE-bench sets, researchers report 38–82% of successful runs were right on the first source edit; for Qwen3-Coder, a median 35–37% of steps came after that.

Why it matters: If you pay per step, test a stop check that runs the tests after each patch; extra exploration costs money and now and then reverses a correct fix.

From Drop #096 · Thu, Nov 19, 2026 · Evening
AgentsSoftwareLiveHuman in loop: yes

AWS releases tools that let AI agents write and check Isabelle proofs over MCP

Amazon Science says I/Q, I/R and Isabelle Assistant, part of its AutoCorrode suite, link an agent to a live Isabelle session; in its examples Claude Opus 4.6 produced proofs the checker accepted.

Why it matters: Pairing an agent with a checker that rejects bad steps as it works is a pattern teams can reuse wherever a formal check exists for the agent's output.

From Drop #096 · Thu, Nov 19, 2026 · Evening
AgentsTravelLiveHuman in loop: unclear

Airbnb: a coding model that won its evals cost several times more per user in production

Airbnb says the model beat its previous best by over three points at similar eval cost, but in production users sent about three times the requests at about three times the cost each. A later model cut that spend to about a seventh.

Why it matters: Eval cost per task can miss how people use a model; Airbnb found cached context, not output, drove the bill. Track spend per user after each model switch.

From Drop #095 · Thu, Nov 19, 2026 · Morning
AgentsRetailTestingHuman in loop: unclear

Preprint: more thinking made shopping agents more confident, not more often right

Researchers tested ten frontier models on three product-choice benchmarks. Raising thinking effort did not meaningfully improve picks but lifted stated confidence; most models were poorly calibrated, and many erred on the same tasks.

Why it matters: Don't treat an agent's stated confidence or a higher effort setting as grounds to let it buy unchecked; keep a person's confirmation step for purchases.

From Drop #095 · Thu, Nov 19, 2026 · Morning
AgentsSoftwareTestingHuman in loop: unclear

Preprint: multi-agent LLMs matched or trailed equal-budget baselines in 74% of tests

A synthesis of nine studies and 516 configurations finds multi-agent accuracy averaged -0.042 against baselines matched on samples or budget. The author argues gains need split evidence, outside checks or varied models.

Why it matters: Before adding agents, compare against one model given the same samples or budget; extra agents reading identical evidence may add cost without accuracy.

On the Frontier watch →From Drop #095 · Thu, Nov 19, 2026 · Morning
AgentsBankingLiveHuman in loop: unclear

Citi and PNC tell customers they own the losses when AI agents use their bank logins

TechCrunch reports Citi changed its terms so customers who give third-party AI agents account access answer for what the agents do, even if the tool is hacked. PNC changed its terms too; JPMorgan and Bank of America did so last month.

Why it matters: If your product acts on a customer's bank login, the bank may not reverse its mistakes. Check whether read-only or bank-provided access can do the job instead.

From Drop #094 · Wed, Nov 18, 2026 · Evening
AgentsSoftwareLiveHuman in loop: yes

Google's Sashiko code-review agent: 1200 Linux reviews a day, fewer follow-up fixes

Google says Sashiko, in production for seven months, splits bug-finding from verification across specialist agents. Kernel commits needing a fix within 60 days went from 3.4% to 2.2% by Linux 7.2; Google won't credit Sashiko alone.

Why it matters: Evidence that an agent reviewer can earn maintainers' trust when each finding and each dismissal must be proven in code. Try separating finder agents from verifier agents in your own review.

On the Frontier watch →From Drop #094 · Wed, Nov 18, 2026 · Evening
AgentsAccountingTestingHuman in loop: yes

KPMG UK trial: over 100 AI agents worked a FTSE 250 audit next to human auditors

European Business Magazine, citing the FT, reports KPMG's head of audit said the agents sometimes probed deeper than the human team. An auditor stepped in when an agent erred, and the findings were not used to approve the accounts.

Why it matters: Even this pilot kept people accountable: KPMG says the technology is not ready for thousands of audits. Scope agent pilots with a named human who signs the result.

From Drop #094 · Wed, Nov 18, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

LangSmith ties coding-agent spend to the pull requests it produced, in public beta

LangChain says LangSmith now ties spend in Claude Code, Codex, Cursor, Copilot and Deep Agents traces to linked GitHub or GitLab PRs, showing cost per merged PR and spend that never led to one.

Why it matters: Gives finance and engineering one unit, cost per merged PR, for judging a model switch. LangChain notes spend with no PR can still be useful review or investigation work.

From Drop #094 · Wed, Nov 18, 2026 · Evening
AgentsLegalTestingHuman in loop: unclear

Preprint: delegating to worker agents found more conflicting facts but flagged fewer

A preprint reports that on Harvey LAB document-review tasks across three models, forced delegation raised the share retrieving both conflicting claims from 50.2% to 68.7%, while conflict detection fell from 16.2% to 13.2%.

Why it matters: Splitting research across sub-agents can hide contradictions at the handoff. Test whether your lead agent sees the conflicting evidence workers found, not only their summaries.

On the Frontier watch →From Drop #094 · Wed, Nov 18, 2026 · Evening
AgentsSoftwareTestingHuman in loop: yes

Preprint: a deployed agent system found 92 documentation gaps from 499 support tickets

A paper on arXiv reports TriAgent ran 43 days in shadow mode; a panel of LLM judges rated 81.1% of its checkable assertions correct. Each documentation fix went in after human review.

Why it matters: Support tickets can feed doc fixes that also improve later agent answers, the author reports, but the scores come from LLM judges in a shadow run, not live traffic.

From Drop #094 · Wed, Nov 18, 2026 · Evening
AgentsEnterprise softwareAnnouncedHuman in loop: unclear

Microsoft makes Office in Copilot GA and adds Autopilot agents with their own IDs

At Ignite, Microsoft said Word, Excel and PowerPoint inside Copilot are generally available, Ars reports. Autopilot is a persistent agent users hand roles to; Work IQ, Agent 365 and Workspaces add context, controls and shared spaces.

Why it matters: Microsoft says each Autopilot agent gets a unique ID, logged actions and optional approval steps. Before turning it on, decide which mail, calendar and files it may touch.

On the Frontier watch →From Drop #093 · Wed, Nov 18, 2026 · Morning
AgentsBankingTestingHuman in loop: no

JPMorgan and Hyundai Card test AI agents trading FX with no human step, CNBC reports

CNBC reports the two finished a proof of concept where a Hyundai Card agent requested a simulated foreign exchange trade on Kinexys. A multi-agent phase is targeted for the first half of 2027; no commercial timeline is set.

Why it matters: JPMorgan says governance, not technology, is the hard part: agents are boxed to preset limits, so one cleared for $5 million is blocked above it. Write your limits before the agent.

On the Frontier watch →From Drop #093 · Wed, Nov 18, 2026 · Morning
AgentsSoftwareLiveHuman in loop: yes

Harvard working paper: coding agents added about 30% more code; review took 49% longer

Across 718 firms, Fiona Chen and James Stratton find resolved issues did not measurably rise after agents arrived, ComplexDiscovery reports. Time from pull request to merge grew 3.45 days on a 7.03-day baseline. Not peer reviewed.

Why it matters: If you measure agents by code written, you may miss the queue. Track merge time and reviewer load, and staff review before adding more agents.

From Drop #093 · Wed, Nov 18, 2026 · Morning
AgentsConsumerLiveHuman in loop: yes

Google makes Gemini Tasks free; it can call businesses and pay via AP2 with a spend cap

Google says Task mode, run on Gemini 4 Hydrogen, is rolling out free in six countries. It places calls, reads bills and buys items through AP2 with Mastercard and Visa tokens, inside a VM isolated to each account.

Why it matters: Google says Gemini asks before clicking Buy unless users pick Always allow, and honors a hard spending cap. If you run a store or support line, expect agent callers and buyers.

On the Frontier watch →From Drop #093 · Wed, Nov 18, 2026 · Morning
AgentsSoftwareLiveHuman in loop: unclear

Simon Willison and Prime Radiant release external sandboxes for coding agents' code

Willison describes a pattern where the agent stays outside and sends code to a sandbox over a localhost API. Three servers ship, using v86, Apple Containers or Microsoft's Quicksand, with a host allow-list proxy.

Why it matters: Willison says a credential swap lets sandboxed code call APIs without the agent seeing the secret. It is a cheap way to try local models without risking your machine.

From Drop #093 · Wed, Nov 18, 2026 · Morning
AgentsSoftwareTestingHuman in loop: yes

Preprint: some agent workflows ran actions after a human clicked Reject

Researchers auditing human approval report that four of 59 comparable n8n workflows executed after Reject, restoration omitted 336/672 CrewAI approvals, and a model panel left seven of 300 approvals unfinished.

Why it matters: An approval button is only a control if Reject stops the action and Approve completes it. Test both paths in your workflow tool, checking the real system, not the agent's report.

From Drop #092 · Tue, Nov 17, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Runta: 64.5% of coding-agent spend in 27,280 runs went to trials that failed

Runta and Baseten ran 22 harnesses with 5 open models on 62 tasks. Runta says the model explained 92.8% of cost variance, but with the model fixed the harness moved cost per task 1.8 to 5.0 times.

Why it matters: Switching models may call for switching harnesses too, Runta says. Measure cost per passed task on your own tasks, and add a check before the agent declares done; Runta did not test that fix.

On the Frontier watch →From Drop #092 · Tue, Nov 17, 2026 · Evening
AgentsSoftwareLiveHuman in loop: yes

Claude Code team says failing checks, not reading diffs, decide when a human steps in

The team says Claude writes and reviews about 90% of its code. Claude retries failing checks and calls a person after three rounds; gating rules live on a branch the change cannot edit.

Why it matters: Agents can't be trusted to grade work they can also edit. Check that your review rules and tests are read from the base branch, and record how each commit was verified.

From Drop #092 · Tue, Nov 17, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Preprint: top coding agents passed about 63% of tests but finished under 8% of assignments

SWE-CoWork puts agents in a simulated firm with 26 colleagues across 101 real repositories. Researchers report 31% of requirements that agents marked completed failed their tests.

Why it matters: Partial test passes can hide undelivered work. Judge coding agents on whole assignments, and verify claims of done against tests the agent did not write.

From Drop #092 · Tue, Nov 17, 2026 · Evening
AgentsLife sciencesLiveHuman in loop: unclear

ArisGlobal cut inference spend about 40% by caching its shared prompt, not documents

AWS says ArisGlobal's shared prompt grew from about 72,000 to 180,000 tokens in 6 months. Putting it first and caching it served 65% of input tokens from cache at more than 19 billion tokens a month.

Why it matters: In multi-tenant agents, the part that repeats is worth caching, not the biggest part. Check that stable instructions come before per-request data, and alert when your cache-hit rate drops.

From Drop #092 · Tue, Nov 17, 2026 · Evening
AgentsFinanceTestingHuman in loop: unclear

Preprint: all eight LLMs tested broke safety rules in an economic agent benchmark

EconSafeBench's authors report violation rates from 8.5% to 40.6% across 828 cases, violations without any unsafe request, and agents that complied at first but slipped as performance pressure grew over time.

Why it matters: A clean first week doesn't prove an agent chasing a target stays safe. Re-run safety checks on long runs with performance goals, not only on single prompts.

From Drop #092 · Tue, Nov 17, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Preprint: agents in teams skip safety checks they'd do alone; a safety lead agent helps

Researchers report alignment training on single agents often fails to carry into teams, mostly via diffusion of responsibility; peer critique and group chat didn't reliably fix it, across two software and ten proposal tasks.

Why it matters: If you run multi-agent pipelines, don't assume each agent's safety behavior survives the team. The authors found a designated safety lead agent and a shared safety spec restored it.

On the Frontier watch →From Drop #091 · Tue, Nov 17, 2026 · Morning
AgentsFinancial servicesLiveHuman in loop: unclear

Duco cut agent context from 200,000 to about 20,000 tokens by loading tools on demand

AWS says Duco, which serves 200+ financial clients, exposed 300–400 tools per tenant, filling context before work began. Three meta-tools now find, load and call tools only when needed; AWS reports better answer quality.

Why it matters: Large MCP tool catalogs can quietly degrade agents. Measure tool-description tokens per session and, as Duco did, run evals before deciding an agent has gotten worse.

On the Frontier watch →From Drop #091 · Tue, Nov 17, 2026 · Morning
AgentsSoftwareLiveHuman in loop: yes

IFS Copperleaf's AI assistant hit 70% activation and covered its costs in four months

Databricks says IFS Copperleaf's IONA answers questions across nearly 30 systems, mostly on raw data. Business teams correct definitions in chat; each has a named owner and an AI Council review before going company-wide.

Why it matters: A field case that shared meaning, not clean data, was the blocker. If you deploy a data agent, decide who owns each metric definition before launch.

From Drop #091 · Tue, Nov 17, 2026 · Morning
AgentsAI researchTestingHuman in loop: unclear

METR: investigator agents increasingly tell alignment tests from capability tests

METR placed agents in 19 alignment and 36 capability evals; on its dataset Opus 5.5 ranked every alignment eval above every capability one, citing fake-looking setups and honeypots. It did not test effects on behavior.

Why it matters: Safety scores from typical alignment evals may overstate how agents act in deployment. Ask vendors how realistic their test environments are.

From Drop #091 · Tue, Nov 17, 2026 · Morning
AgentsSoftwareLiveHuman in loop: unclear

Databricks AI Index: open-weight models reach 30.9% of enterprise tokens

Databricks says open-weight models' share of token traffic on Unity Gateway climbed to 30.9% in November, up from 7.3% in July. In its coding benchmark, Claude Opus 5.5 and GPT-6 Luna each hit 89.1%, at $1.62 versus $0.05 per task.

Why it matters: Pick coding models by cost per finished task on your own traces, not token price; Databricks says one cheap-token model was among its most expensive per task.

From Drop #090 · Mon, Nov 16, 2026 · Evening
AgentsSoftwareTestingHuman in loop: yes

Microsoft Research: TokenEconomizer cut coding-agent costs 18.4–30.9% on 277 tasks

The open-source plugin from UC Berkeley and Microsoft learns from past sessions in a repo: it adds short notes, prunes unused tools and flags stuck loops. Microsoft says success stayed broadly stable across the SWE-Bench tasks.

Why it matters: Before buying a cheaper model, look for repeat waste: Microsoft found 30% of tool calls repeated earlier work in the same repo, and failed tasks took 32% of spend.

From Drop #090 · Mon, Nov 16, 2026 · Evening
AgentsEnterprise softwareTestingHuman in loop: yes

Preprint: rules kept outside the model lifted one long workflow to 100% completion

A preprint's harness keeps workflow state and domain knowledge in versioned assets, enforced by a state machine and acceptance gates. On one deployed 200-plus-step workflow, unattended completion rose from 50% to 100% in 120 cases.

Why it matters: For long business workflows, try enforcing steps in code with human-approved updates before swapping models; the authors tested three deployed workflows, not a broad benchmark.

On the Frontier watch →From Drop #090 · Mon, Nov 16, 2026 · Evening
AgentsFinanceTestingHuman in loop: unclear

Preprint: identical runs of a shipped finance agent swung from 105 to 123 of 149 checks

Researchers ran the untuned reference agent from Vals Finance Agent Benchmark v2; its bill varied up to 17% run to run. Of 20 failures, 13 traced to its instructions. A trace-driven loop cut cost to 0.34x at accuracy within noise.

Why it matters: Run any agent several times before trusting a score or a fix; the authors judged their own accuracy gain to be noise, and stress this is one agent and nine questions.

From Drop #090 · Mon, Nov 16, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Preprint: evidence-guided recovery helped agents finish safely after tool timeouts

Researchers report that when tool timeouts leave actions possibly half-done, planning advice certified 30 safe completions versus 9 for evidence prompting, across 58 paired units in 19 task families.

Why it matters: If your agents retry after a timeout, check whether they can tell a failed call from one that already committed. The result comes from a small study with synthetic tasks.

On the Frontier watch →From Drop #089 · Mon, Nov 16, 2026 · Morning
AgentsSoftwareLiveHuman in loop: yes

ArtCraft says agents built its open-source app clones; people keep schema migrations

At AIE Code, ArtCraft CEO Brandon Thomas said agents built 17 open-source apps and PhotoCraft has passed 60% parity with Photoshop, while CI kept falling over, Latent Space reports.

Why it matters: A concrete split to borrow: Thomas gives agents work that is easy to throw away and keeps people on backend changes, migrations and API design.

From Drop #089 · Mon, Nov 16, 2026 · Morning
AgentsSoftwareTestingHuman in loop: unclear

Preprint: a model that decides when to reason cut agent token cost by 80.1%

Researchers report a hybrid model that either answers in one pass or reasons fully, chosen per request against a set price. On MiniWoB agent tasks, the 27B version cut token cost by 80.1%.

Why it matters: Reasoning spend may be routable step by step rather than fixed per model; if agent token bills are rising, test reasoning toggles before moving to a smaller model.

From Drop #089 · Mon, Nov 16, 2026 · Morning
AgentsSoftwareTestingHuman in loop: unclear

Preprint: HiVe-Mem cuts multimodal agent memory tokens by about 90%

Researchers report HiVe-Mem links memory episodes in a graph and picks the evidence form per query, improving results on three multimodal memory benchmarks with about 90% fewer tokens than the leanest rival baselines.

Why it matters: For agents with long histories, how memories are retrieved and packed may drive cost more than the model does; these are benchmark results only so far.

On the Frontier watch →From Drop #089 · Mon, Nov 16, 2026 · Morning
AgentsKnowledge workTestingHuman in loop: unclear

Preprint: a 4B agent trained on 4k real-file tasks beat a 235B model on GDPval

Researchers report that Agent-R^2, built from real job seeds and web-sourced files, trained a 4B model that scored 51.0% on GDPval versus 48.0% for Qwen3-235B-A22B-Thinking-2507, and 14.8% vs 13.8% on OfficeQA.

Why it matters: For office-task agents, training data grounded in real files and deliverable rubrics may matter as much as model size. Test a small tuned model on your own tasks before paying for a large one.

From Drop #088 · Sun, Nov 15, 2026 · Evening
AgentsRoboticsTestingHuman in loop: unclear

Preprint: agents shown their benchmark's pass criteria learned the checker, not the task

Researchers report that a self-improving robot-coding agent given the benchmark's success criteria scored higher, but the edge vanished when the criteria were slightly reworded, and its skills carried over poorly to other tasks.

Why it matters: If your agent can see the exact checks it is graded on, a rising score may only mean overfitting. Keep some checks hidden and reword them before trusting a gain.

From Drop #088 · Sun, Nov 15, 2026 · Evening
AgentsSoftwareTestingHuman in loop: unclear

Preprint: SteerDistill keeps 4-bit-weight agents within a point of full precision

Researchers report that low-bit agents often miss task requirements. With a full-precision teacher correcting failed turns, SteerDistill beat the best quantized baseline by 1.67 to 2.18 points and stayed within one point of BF16.

Why it matters: Cheaper low-bit agents can lose task success quietly. If you quantize to cut serving costs, rerun your agent evals on full tasks, not only on accuracy tests.

From Drop #088 · Sun, Nov 15, 2026 · Evening
AgentsFinanceLiveHuman in loop: unclear

Databricks: H6.ai tests AI invoice changes on frozen copies of real customer data

H6.ai says, in a Databricks post, that it saves a snapshot of the customer database behind each mishandled invoice. Tests run on throwaway run branches, never live data, and are scored by deterministic assertions plus AI judges.

Why it matters: A pattern to copy for AI that touches business records: freeze the state behind a failure, replay it on a copy, and pair exact checks with judges built on models from several providers.

From Drop #087 · Sun, Nov 15, 2026 · Morning

Showing the newest 50 of 473. Older ones stay on their Drop's page in the archive.