Updated
Updated · KDnuggets · Nov 17
Scikit-learn Tutorial Fixes 0.9 Probability Errors With 2 Calibration Methods
Updated
Updated · KDnuggets · Nov 17

Scikit-learn Tutorial Fixes 0.9 Probability Errors With 2 Calibration Methods

2 articles · Updated · KDnuggets · Nov 17

Summary

  • Scikit-learn’s latest tutorial shows how to test whether predicted probabilities are trustworthy, then recalibrate them with Platt scaling and isotonic regression.
  • On a 100,000-row Covertype sample with 54 features, it trains Gaussian Naive Bayes and a 200-tree random forest, then uses reliability diagrams, Brier score and a 0.85-0.95 “ninety test” to expose over- and under-confidence.
  • A 3-way split—50% train, 20% calibration, 30% test—is central because calibrators must learn on data independent of the classifier’s training set; the guide notes scikit-learn 1.6+ and FrozenEstimator replace older cv="prefit" workflows.
  • Platt scaling fits a 2-parameter sigmoid and is presented as safer for small calibration sets or when ROC-AUC ranking must stay unchanged, while isotonic regression is more flexible but can overfit and may alter ROC-AUC slightly through tied scores.
  • The tutorial’s rule of thumb is to prefer sigmoid below about 1,000 calibration samples, compare isotonic once you have several thousand, and use CalibratedClassifierCV with cross-validation when no separate calibration set is available.

Insights

Scikit-learn updated its workflow, but are post-hoc fixes truly better than building inherently calibrated algorithms from the start?
Why do top data scientists sacrifice valuable training data just to fix a hidden flaw in machine learning probabilities?
Could your highly accurate AI model be secretly overconfident and costing your business money without you realizing it?