On the backward pass, significant additional latency savings come from skipping entries that quantize to zero. Table A1 shows how helpful this pruning is.

I know there are a lot of dataheads on bluesky right now: here is my first draft of an answer to “What is the typical approach for storing large text datasets in the context of LM pretraining (and shouldn’t you use a relational db)”. If anyone has any deep thoughts or hot takes definitely come at me

I agree and this often feels really confusing and wrong to data engineers. Why? Too long for a reply so it’s an image. Thank you for your work