Scikit-learn Tutorial Fixes 0.9 Probability Errors With 2 Calibration Methods
Updated
Updated · KDnuggets · Nov 17
Scikit-learn Tutorial Fixes 0.9 Probability Errors With 2 Calibration Methods
2 articles · Updated · KDnuggets · Nov 17
Summary
Scikit-learn’s latest tutorial shows how to test whether predicted probabilities are trustworthy, then recalibrate them with Platt scaling and isotonic regression.
On a 100,000-row Covertype sample with 54 features, it trains Gaussian Naive Bayes and a 200-tree random forest, then uses reliability diagrams, Brier score and a 0.85-0.95 “ninety test” to expose over- and under-confidence.
A 3-way split—50% train, 20% calibration, 30% test—is central because calibrators must learn on data independent of the classifier’s training set; the guide notes scikit-learn 1.6+ and FrozenEstimator replace older cv="prefit" workflows.
Platt scaling fits a 2-parameter sigmoid and is presented as safer for small calibration sets or when ROC-AUC ranking must stay unchanged, while isotonic regression is more flexible but can overfit and may alter ROC-AUC slightly through tied scores.
The tutorial’s rule of thumb is to prefer sigmoid below about 1,000 calibration samples, compare isotonic once you have several thousand, and use CalibratedClassifierCV with cross-validation when no separate calibration set is available.