Fabian Schaipp
@fschaipp.bsky.social
Researcher in Optimization for ML at Inria Paris. Previously at TU Munich.
https://fabian-sp.github.io/
https://fabian-sp.github.io/
🚋 New blog post:
On "infinite" learning-rate schedules and how to construct them from one checkpoint to the next.
fabian-sp.github.io/posts/2025/0...
On "infinite" learning-rate schedules and how to construct them from one checkpoint to the next.
fabian-sp.github.io/posts/2025/0...
Infinite Schedules and the Benefits of Lookahead
TL;DR: Knowing the next training checkpoint in advance (“lookahead”) helps to set the learning rate. In the limit, the classical square-root schedule appears on the horizon.
fabian-sp.github.io
September 1, 2025 at 11:15 AM
🚋 New blog post:
On "infinite" learning-rate schedules and how to construct them from one checkpoint to the next.
fabian-sp.github.io/posts/2025/0...
On "infinite" learning-rate schedules and how to construct them from one checkpoint to the next.
fabian-sp.github.io/posts/2025/0...
Learning rate schedules seem mysterious? Why is the loss going down so fast during cooldown?
Turns out that this behaviour can be described with a bound from *convex, nonsmooth* optimization.
A short thread on our latest paper 🚞
arxiv.org/abs/2501.18965
Turns out that this behaviour can be described with a bound from *convex, nonsmooth* optimization.
A short thread on our latest paper 🚞
arxiv.org/abs/2501.18965
The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training
We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedul...
arxiv.org
February 5, 2025 at 10:13 AM
Learning rate schedules seem mysterious? Why is the loss going down so fast during cooldown?
Turns out that this behaviour can be described with a bound from *convex, nonsmooth* optimization.
A short thread on our latest paper 🚞
arxiv.org/abs/2501.18965
Turns out that this behaviour can be described with a bound from *convex, nonsmooth* optimization.
A short thread on our latest paper 🚞
arxiv.org/abs/2501.18965
That time of the year again, where you delete a word and latex manages to make the line <longer>.
January 24, 2025 at 9:26 AM
That time of the year again, where you delete a word and latex manages to make the line <longer>.
Want all NeurIPS/ICML/ICLR papers in one single .bib file? Here you go!
🗞️ short blog post: fabian-sp.github.io/posts/2024/1...
📇 bib files: github.com/fabian-sp/ml-bib
🗞️ short blog post: fabian-sp.github.io/posts/2024/1...
📇 bib files: github.com/fabian-sp/ml-bib
A Bibliography Database for Machine Learning
Getting the correct bibtex entry for a conference paper (e.g. published at NeurIPS, ICML, ICLR) is annoyingly hard: if you search for the title, you will often find a link to arxiv or to the pdf file,...
fabian-sp.github.io
December 17, 2024 at 10:42 AM
Want all NeurIPS/ICML/ICLR papers in one single .bib file? Here you go!
🗞️ short blog post: fabian-sp.github.io/posts/2024/1...
📇 bib files: github.com/fabian-sp/ml-bib
🗞️ short blog post: fabian-sp.github.io/posts/2024/1...
📇 bib files: github.com/fabian-sp/ml-bib
Reposted by Fabian Schaipp
Generating cat videos is nice, but what if you could tackle real scientific problems with the same methods? 🧪🌌
Introducing The Well: 16 datasets (15TB) for Machine Learning, from astrophysics to fluid dynamics and biology.
🐙: github.com/PolymathicAI...
📜: openreview.net/pdf?id=00Sx5...
Introducing The Well: 16 datasets (15TB) for Machine Learning, from astrophysics to fluid dynamics and biology.
🐙: github.com/PolymathicAI...
📜: openreview.net/pdf?id=00Sx5...
December 2, 2024 at 4:08 PM
Generating cat videos is nice, but what if you could tackle real scientific problems with the same methods? 🧪🌌
Introducing The Well: 16 datasets (15TB) for Machine Learning, from astrophysics to fluid dynamics and biology.
🐙: github.com/PolymathicAI...
📜: openreview.net/pdf?id=00Sx5...
Introducing The Well: 16 datasets (15TB) for Machine Learning, from astrophysics to fluid dynamics and biology.
🐙: github.com/PolymathicAI...
📜: openreview.net/pdf?id=00Sx5...
Not so fun exercise: take a recent paper that you consider exceptionally good, and one that you think is mediocre (at best).
Then look up their reviews on ICLR 2025. I find these reviews completely arbitrary most of the times.
Then look up their reviews on ICLR 2025. I find these reviews completely arbitrary most of the times.
November 25, 2024 at 3:40 PM
Not so fun exercise: take a recent paper that you consider exceptionally good, and one that you think is mediocre (at best).
Then look up their reviews on ICLR 2025. I find these reviews completely arbitrary most of the times.
Then look up their reviews on ICLR 2025. I find these reviews completely arbitrary most of the times.
my French 🇨🇵 digital bank (supposedly!) today asked me (via letter) to confirm an account action via sending them a signed letter. wtf
November 25, 2024 at 12:41 PM
my French 🇨🇵 digital bank (supposedly!) today asked me (via letter) to confirm an account action via sending them a signed letter. wtf
Reposted by Fabian Schaipp
I made a #starterpack for computational math 💻🧮 so please
1. share
2. let me know if you want to be on the list!
(I have many new followers which I do not know well yet, so I'm sorry if you follow me and are not on here, but want to - drop me a note and I'll add you!)
go.bsky.app/DXdZkzV
1. share
2. let me know if you want to be on the list!
(I have many new followers which I do not know well yet, so I'm sorry if you follow me and are not on here, but want to - drop me a note and I'll add you!)
go.bsky.app/DXdZkzV
November 18, 2024 at 3:08 PM
I made a #starterpack for computational math 💻🧮 so please
1. share
2. let me know if you want to be on the list!
(I have many new followers which I do not know well yet, so I'm sorry if you follow me and are not on here, but want to - drop me a note and I'll add you!)
go.bsky.app/DXdZkzV
1. share
2. let me know if you want to be on the list!
(I have many new followers which I do not know well yet, so I'm sorry if you follow me and are not on here, but want to - drop me a note and I'll add you!)
go.bsky.app/DXdZkzV