Airbnb: a coding model that won its evals cost several times more per user in production
Airbnb says the model beat its previous best by over three points at similar eval cost, but in production users sent about three times the requests at about three times the cost each. A later model cut that spend to about a seventh.
Why it matters: Eval cost per task can miss how people use a model; Airbnb found cached context, not output, drove the bill. Track spend per user after each model switch.
From Drop #095 · Thu, Nov 19, 2026 · Morning