What we show:
1️⃣ L1/L2/no regularization destroys perfectly generalizing solutions across 6 formal-language tasks
2️⃣ MDL keeps or improves on these same perfect solutions

(6/7)

Imagine memorizing an entire book using just one weight—by encoding the book as a binary string and placing it after a decimal point (if assuming very large / infinite precision).

Tiny weight, full memorization.

(3/7)