TFDS or github.com/google/grain. I personally mainly use TFDS as it lowers to C++ so I'm able to squeeze much more out of it. I'm using the more functional and random-access tfds.data_source API which uses the ArrayRecord format internally.

We invented nukes so the good guys could uphold world peace and ensure they're never used.

Also, when I mention entropy, I can't help but think of @veritasium-science.bsky.social's excellent video, which really deepened my intuition about entropy and related concepts, especially free energy.

One nice thing about being able to choose C is that it essentially allows us to swap the log basis, since it's just a division with a different constant.

Yes, I believe so. The larger model is trained via next-token prediction and the smaller ones are distilled.

Gemma 2 models use knowledge distillation.

Pretty much "the usual", though it feels uncomfortably close lately.

You can commission a formal search for a few hundred pounds, but if it shows you're liable, you lose the option to get insurance. Since the insurance costs just tens of pounds, it’s better to skip the search and go straight for the insurance. So "don’t ask, don’t tell" is the smarter choice!

Andrew and Gail Wallbank were billed nearly £100,000 to repair their parish’s medieval church in Aston Cantlow, Warwickshire. After a long legal battle, the Law Lords sided with the church, leaving the couple with a £350,000 total bill, including legal costs.

I wonder why he left Bluesky and went back to the dark side.