Signal SentrySearchUpdated Nov 20, 05:30 UTCAM Drop
Agents · industry

Agents in Data engineering

Agent deployments and launches in Data engineering, from the Drops.

From the Drops

AgentsData engineeringTestingHuman in loop: yes

Databricks: over half of 80,000-plus pipeline issues needed in-house knowledge to fix

Databricks says its ZeroOps pipeline-repair agent saw over 80,000 customer issues at hundreds of companies in preview. Now in public beta, it adds custom instructions and one-click pull requests, and a person reviews every fix first.

Why it matters: Generic agents miss house rules. Before judging any ops agent, budget time to write down your flaky sources, preferred fixes and naming conventions for it.

From Drop #097 · Fri, Nov 20, 2026 · Morning
AgentsData engineeringLiveHuman in loop: unclear

Databricks: agents building declarative Spark pipelines used ~34x less compute

Databricks says agents now create over 70% of its declarative (SDP) pipelines. In 528 builds on 22 tasks with Opus 4.8, procedural runs used 596 executor-seconds vs 17 for declarative, with 51% less code.

Why it matters: Declarative builds cost about 2.3x more in tokens but catch structural errors before a cluster runs. Databricks says logic errors still need tests and review.

From Drop #044 · Sat, Oct 24, 2026 · Evening