James Shore Urges Randomized AI Trials to Measure Software Speed, Citing 50% Benefit Overestimates
Updated
Updated · O'Reilly Media · Nov 4
James Shore Urges Randomized AI Trials to Measure Software Speed, Citing 50% Benefit Overestimates
1 articles · Updated · O'Reilly Media · Nov 4
Summary
James Shore says companies should use a randomized controlled trial—not PR counts, story points or estimates alone—to measure whether AI actually speeds software delivery.
His proposed design compares each developer or team to itself: record an estimate before AI use is known, then randomly assign tasks from a 50-50 with-AI and without-AI pool.
That setup is meant to cut noise from differing work styles, shifting estimates and parallel process changes, while collecting estimated time, actual time, task type, spending and rework data.
Shore argues the rigor matters because AI costs can range from hundreds to thousands of dollars per person per month, and a METR study found engineers overestimated speed gains by nearly 50%.
He also warns that a 2x coding-speed gain may translate to only 1.33x overall improvement if developers spend 50% of their week on other work, with broader productivity and side effects left for later parts of the series.