Data Scientists Get 5-Step Playbook for Directing AI Agents

Q: Why did a Bitcoin fraud model’s F1 score fall from 0.87 to 0.70 once humans enforced real-world rules the AI missed?

The score fell because the original 0.87 result was inflated by two classic modeling mistakes the AI agent was not told to guard against: data leakage and unrealistic evaluation. When asked only to “build a model,” the agent used a random train-test split on time-ordered Bitcoin transactions. That let information from later transactions bleed into training, effectively giving the model a look at the future. In a fraud system meant to predict new cases, that is not how the model would be used in production. The agent also relied on a planted feature that acted as a proxy for the fraud label, another form of leakage that can make a model look far stronger than it really is. Once humans imposed real-world rules, the test became much harder and more honest. They required a temporal holdout so the model trained on earlier transactions and was tested on later ones, removed the leaking feature, and checked performance on high-degree nodes that matter operationally. Under those conditions, the model could no longer benefit from future information or shortcut signals, so its F1 score dropped to 0.70. That decline was the point of the exercise: the lower score was more credible. It showed the earlier metric was measuring an easy, contaminated benchmark rather than real predictive power. The sharper drop in recall, especially to 0.21 on high-degree nodes, also showed why aggregate metrics alone can hide weaknesses that matter in deployment.

3 articles · Updated · O'Reilly Media · Sep 30

Five practices anchor the playbook: frame the investigation, equip the agent, organize experiments, review results independently, and turn lessons into reusable skills and evals.
A fraud-detection test showed why that structure matters: Claude first posted F1 of 0.87 and ROC AUC of 0.99 using a random split and a planted leaking feature, overstating real-world performance.
After the team enforced a temporal holdout, removed leakage and required subgroup reporting, F1 fell to 0.70, overall recall to 0.61 and high-degree-node recall to 0.21.
The article argues data scientists should shift from manually executing every step to specifying the question, runtime and constraints, then verifying evidence through bounded experiments or competing analyses.
That workflow extends to reusable infrastructure—skills, access controls, evidence trails and independent review—especially when colleagues use agents directly rather than through a human analyst.