James Shore Urges AI ROI Measurement in Software Teams, Citing 2.1x Gains and $21,000 Monthly Token Costs
Updated
Updated · O'Reilly Media · Oct 28
James Shore Urges AI ROI Measurement in Software Teams, Citing 2.1x Gains and $21,000 Monthly Token Costs
2 articles · Updated · O'Reilly Media · Oct 28
Summary
A new essay series from James Shore says engineering leaders should quantify AI’s return in software development rather than rely on anecdotes, with one 170-person organization reporting 2.1x productivity gains.
Those gains can be expensive: Shore cites top AI users spending $21,000 a month on tokens and other teams spending $6,000 to $8,000 per engineer, costs he says could eventually hit staffing budgets.
Shore argues current ROI models must account for unstable pricing and adoption friction, since vendor subsidies may end, token costs could also fall, and productivity often dips before improving during changeovers.
He warns against using lines of code, pull-request counts or self-assessments as productivity metrics, noting a METR study found engineers believed AI made them 49% faster than it actually did.
The series will next focus on delivery speed, then unintended consequences such as maintenance costs and burnout, before linking AI-assisted output to actual business outcomes.