ChatGPT Repeats 4 Data Analysis Errors Across 3 Datasets, Missing Them in Self-Review
Updated
Updated · KDnuggets · Sep 3
ChatGPT Repeats 4 Data Analysis Errors Across 3 Datasets, Missing Them in Self-Review
3 articles · Updated · KDnuggets · Sep 3
Summary
An experiment on 3 small business-style datasets found ChatGPT made four recurring analysis mistakes, and a review pass corrected only one category while approving or worsening the others.
The failures came from choosing the wrong metric, writing prose numbers that code never produced, inferring a false speedup from 18 undelivered orders, and ignoring 226 missing height values that shrank one sample by 64%.
On shipment data, the model reported 2.6 days by measuring shipped-to-delivered instead of the asked ordered-to-delivered time of 6.09 days, then called week-3 shipping faster even though 7 of 10 orders were still unresolved.
On sales and Olympics files, it inflated APAC sales to "$3.68M" from 3675.49 with no computed basis, and its self-review invented a new height comparison that reversed the original conclusion without running code.
The test found ChatGPT handled mechanics such as pandas, SQL and date parsing well, but still failed when the right answer depended on context outside the visible table or on what the data omitted.
If AI confidently invents numbers and ignores missing data, are businesses risking millions by blindly trusting these models for executive reporting?
Can future AI architectures bridge the gap between generating plausible text and truly understanding the complex operational reality behind business data?