LightNews LightNews
banner
eify.bsky.social
@eify.bsky.social
15 followers 13 following 9 posts
Posts Replies Media Videos
eify.bsky.social
It seems that AdamW & MARS (arxiv.org/abs/2411.10438) effectively reach the same val loss for GPT-2 small with the optimal LR according to Appendix B, in contrast to Figure 1? @quanquangu.bsky.social

If MARS is less sensitive to LR that's also an advantage, but a different kind.
Figure 6: Validation loss with respect to training tokens for AdamW with learning rates 6 × 10−4, 3 × 10−3 and MARS with learning rate 3 × 10−3 on GPT-2 small model (125M). The model trained with MARS achieves a validation loss of 2.852. Figure 1: The training and validation loss curves, plotted against both training tokens and wall-clock time on GPT-2 small model (125M).
December 4, 2024 at 9:31 PM
eify.bsky.social
Strange takeaway from that article, I have to say...
All the US have to do to help is to cut Chinese cleantech tariff.
www.theguardian.com/business/art...
The US is still the biggest contributor to global warming - by far
November 28, 2024 at 9:42 PM