Olivier Grisel
LightNews LightNews
banner
ogrisel.bsky.social
Olivier Grisel
@ogrisel.bsky.social
2.6K followers 1.1K following 100 posts
Software engineer at probabl, scikit-learn contributor.

Also at:

https://sigmoid.social/@ogrisel
https://github.com/ogrisel
Posts Replies Media Videos
ogrisel.bsky.social
We then discussed another common related problem: how to deal with a prevalence shift between observed data and the deployment setting?

probabl-ai.github.io/calibration-...
Diagram explaining the general flow of data science operations to correct for prevalence shifts. See the linked notebook for an exhaustive description of the setting.
August 19, 2025 at 11:58 AM
ogrisel.bsky.social
Attending the @skrub-data.bsky.social tutorial by @riccardocappuzzo.com and @glemaitre58.bsky.social at #EuroScipy2025. They introduce the new DataOps feature released in skrub 0.6.

Here is the repo with the material for the tutorial: github.com/skrub-data/E...
Photo of Riccardo presenting skrub DataOps in a lecture room to an audience of ~50 people.
August 18, 2025 at 9:08 AM
ogrisel.bsky.social
Recently merged in scikit-learn's main branch: display the maximum predicted class probability in 2D continuous feature spaces (mostly for didactic purposes):

scikit-learn.org/dev/auto_exa...

The linked example has been updated to include some conclusions we can draw from this plot.
Outpout on DecisionBoundaryDisplay for a set of probabilistic classifier on a 3 class classification problems. The two logistic regression models fitted on the original features display linear decision boundaries as expected. For this particular problem, this does not seem to be detrimental as both models are competitive with the non-linear models when quantitatively evaluated on the test set. We can observe that the amount of regularization influences the model confidence: lighter colors for the strongly regularized model with a lower value of C. Regularization also impacts the orientation of decision boundary leading to slightly different ROC AUC. The log-loss on the other hand evaluates both sharpness and calibration and as a result strongly favors the weakly regularized logistic-regression model, probably because the strongly regularized model is under-confident. This could be confirmed by looking at the calibration curve using sklearn.calibration.CalibrationDisplay. The logistic regression model with RBF features has a “blobby” decision boundary that is non-linear in the original feature space and is quite similar to the decision boundary of the Gaussian process classifier which is configured to use an RBF kernel. The logistic regression model fitted on binned features with interactions has a decision boundary that is non-linear in the original feature space and is quite similar to the decision boundary of the gradient boosting classifier: both models favor axis-aligned decisions when extrapolating to unseen region of the feature space. The logistic regression model fitted on spline features with interactions has a similar axis-aligned extrapolation behavior but a smoother decision boundary in the dense region of the feature space than the two previous models.
March 7, 2025 at 10:58 AM
ogrisel.bsky.social
I recently shared some of my reflections on how to use probabilistic classifiers for optimal decision-making under uncertainty at @pydataparis.bsky.social 2024.

Here is the recording of the presentation:

www.youtube.com/watch?v=-gYn...
A high-level summary diagram taken from the slides linked below. It shows the interplay of two main components: a probabilistic model and decision maker or planner. Probabilistic predictions of an underfitting polynomial classifier on a noisy XOR task and the corresponding under-confident calibration curve. Probabilistic predictions of an overfitting polynomial classifier and the resulting overconfident calibration curve on the same noisy XOR problem. Simulation study to show the relative lack of stability of hyperparameter tuning when using hard metrics such as Accuracy or soft yet not probabilistic metrics such as ROC AUC compared to a strictly proper scoring rule such as the log-loss.
November 27, 2024 at 2:17 PM
ogrisel.bsky.social
But empirical results in the linked paper seem to indicate that it does help reduce the expected calibration error (ECE) when the training set is large enough w.r.t. the model complexity, in particular when the models are trained with Label Smoothing or post-processed with Temperature Scaling.
Screenshot of various ECE measurements for different model sizes, datasets and gamma values, with and without Psi^gamma correction.
November 20, 2024 at 9:38 AM
ogrisel.bsky.social
Furthermore, there exists a closed-form formula of a function (named Psi^gamma in the paper) to recover the true conditional probabilities from the conditional scores of the FL minimizer:
Screenshot of the math formula. Here is the latex transcript of the main equations: \begin{equation} \eta(\boldsymbol{x}) = \boldsymbol{\Psi}^{\gamma}(\boldsymbol{q}^{\gamma,*}(\boldsymbol{x})), \end{equation} where \begin{equation} \boldsymbol{\Psi}^{\gamma}(\boldsymbol{v}) = \left[\Psi_{1}^{\gamma}(\boldsymbol{v}), \ldots, \Psi_{K}^{\gamma}(\boldsymbol{v})\right]^{\top}, \end{equation} \begin{equation} \Psi_{i}^{\gamma}(\boldsymbol{v}) = \frac{h^{\gamma}(v_i)}{\sum_{l=1}^{K} h^{\gamma}(v_l)}, \end{equation} \begin{equation} h^{\gamma}(v) = \frac{v}{\varphi^{\gamma}(v)} = \frac{v}{(1-v)^{\gamma} - \gamma(1-v)^{\gamma-1}v \log v}. \end{equation} Extracted using https://chat.mistral.ai.
November 20, 2024 at 9:38 AM
ogrisel.bsky.social
The Focal Loss is very popular for imbalanced multiclass computer vision tasks, but it is not a strictly proper scoring rule: its minimizer can be under or overconfident.

However, taking the argmax-based predictions of its minimizer recovers the Bayes optimal classifier.

arxiv.org/abs/2011.09172
Graph showing under and overconfidence of FL minimizers for different values of gamma. The x-axis represents the max of the true conditional class probabilities, while the y-axis represents the maximum conditional prediction of the FL minimizer. For binary problems, the FL minimizer is more and more under confident for larger and larger values of gamma, while it can be either over or under confident for 1000-way classification problems.
November 20, 2024 at 9:38 AM