bourbaki7
LightNews LightNews
banner
bourbaki7.bsky.social
bourbaki7
@bourbaki7.bsky.social
82 followers 150 following 87 posts
Insurance quant, posting AI papers
Posts Replies Media Videos
Reposted by bourbaki7
tylerhuckabee.bsky.social
mtg spent like ten years making enemies on the left and that seemed fine with her. she spent ten days making enemies on the right, had to hire security, and then had to quit altogether.
November 22, 2025 at 4:19 AM
Reposted by bourbaki7
walmsley.bsky.social
unironically a bad sign for the future of the country that she's been pressured in to doing this
kylegriffin1.bsky.social Kyle Griffin @kylegriffin1.bsky.social · 3d
BREAKING: Marjorie Taylor Greene says she's resigning from Congress — effective January 5, 2026.
November 22, 2025 at 1:24 AM
Reposted by bourbaki7
utopia-defer.red
Yoda more olives meme. Top panel: after posting 56,000 times straight while on bluesky at 11:00am Bottom panel: 11:08am... Netanyahu posted a Jacobin article... going back for more posting
November 21, 2025 at 5:13 PM
Reposted by bourbaki7
nytpitchbot.bsky.social
Sources close to Jared and Ivanka say that, privately, the couple disapproves of executing the president's political opponents.
November 21, 2025 at 4:23 AM
Reposted by bourbaki7
beyerstein.bsky.social
Man notorious for fucking teenagers in Russia thinks the Epstein scandal is overblown.
noupside.bsky.social Renee DiResta @noupside.bsky.social · 5d
Ok Matt Taibbi has posted.
November 20, 2025 at 4:52 AM
bourbaki7.bsky.social
arxiv.org/abs/2509.20328
Video models are zero-shot learners and reasoners
The remarkable zero-shot capabilities of Large Language Models (LLMs) have propelled natural language processing from task-specific models to unified, generalist foundation models. This transformation...
arxiv.org
September 26, 2025 at 5:53 AM
bourbaki7.bsky.social
arxiv.org/abs/2509.17567
LIMI: Less is More for Agency
We define Agency as the emergent capacity of AI systems to function as autonomous agents actively discovering problems, formulating hypotheses, and executing solutions through self-directed engagement...
arxiv.org
September 25, 2025 at 3:37 AM
bourbaki7.bsky.social
arxiv.org/abs/2509.14786
Pre-training under infinite compute
Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We first show that exi...
arxiv.org
September 21, 2025 at 10:57 PM
bourbaki7.bsky.social
arxiv.org/abs/2509.10167
The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagrams
We study the gradient-based training of large-depth residual networks (ResNets) from standard random initializations. We show that with a diverging depth $L$, a fixed embedding dimension $D$, and an a...
arxiv.org
September 21, 2025 at 3:20 AM
bourbaki7.bsky.social
arxiv.org/abs/2509.07301
Causal Attention with Lookahead Keys
In standard causal attention, each token's query, key, and value (QKV) are static and encode only preceding context. We introduce CAuSal aTtention with Lookahead kEys (CASTLE), an attention mechanism ...
arxiv.org
September 21, 2025 at 3:18 AM
bourbaki7.bsky.social
Not a new paper, but I hadn't seen it til now

arxiv.org/abs/2411.16085
Cautious Optimizers: Improving Training with One Line of Code
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we...
arxiv.org
July 26, 2025 at 5:47 AM
bourbaki7.bsky.social
arxiv.org/abs/2507.02754
Fast and Simplex: 2-Simplicial Attention in Triton
Recent work has shown that training loss scales as a power law with both model size and the number of tokens, and that achieving compute-optimal models requires scaling model size and token count toge...
arxiv.org
July 26, 2025 at 5:25 AM
bourbaki7.bsky.social
arxiv.org/abs/2507.10613
Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs
Traditional scaling laws in natural language processing suggest that increasing model size and training data enhances performance. However, recent studies reveal deviations, particularly in large lang...
arxiv.org
July 26, 2025 at 5:22 AM
bourbaki7.bsky.social
arxiv.org/abs/2507.13338
Training Transformers with Enforced Lipschitz Constants
Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, divergent training, and ove...
arxiv.org
July 23, 2025 at 9:16 PM
bourbaki7.bsky.social
arxiv.org/abs/2507.09404
Scaling Laws for Optimal Data Mixtures
Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard appro...
arxiv.org
July 19, 2025 at 11:30 PM
bourbaki7.bsky.social
arxiv.org/abs/2507.06261
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
In this report, we introduce the Gemini 2.X model family: Gemini 2.5 Pro and Gemini 2.5 Flash, as well as our earlier Gemini 2.0 Flash and Flash-Lite models. Gemini 2.5 Pro is our most capable model y...
arxiv.org
July 14, 2025 at 3:24 AM
bourbaki7.bsky.social
arxiv.org/abs/2507.07955
Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
Despite incredible progress in language models (LMs) in recent years, largely resulting from moving away from specialized models designed for specific tasks to general models based on powerful archite...
arxiv.org
July 13, 2025 at 4:49 PM
bourbaki7.bsky.social
arxiv.org/abs/2506.19697
Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
Extreme activation outliers in Large Language Models (LLMs) critically degrade quantization performance, hindering efficient on-device deployment. While channel-wise operations and adaptive gradient s...
arxiv.org
July 13, 2025 at 7:59 AM
bourbaki7.bsky.social
arxiv.org/abs/2506.19143
Thought Anchors: Which LLM Reasoning Steps Matter?
Reasoning large language models have recently achieved state-of-the-art performance in many fields. However, their long-form chain-of-thought reasoning creates interpretability challenges as each gene...
arxiv.org
July 13, 2025 at 3:01 AM
bourbaki7.bsky.social
arxiv.org/abs/2505.21487
Hardware-Efficient Attention for Fast Decoding
LLM decoding is bottlenecked for large batches and long contexts by loading the key-value (KV) cache from high-bandwidth memory, which inflates per-token latency, while the sequential nature of decodi...
arxiv.org
July 10, 2025 at 9:36 PM
bourbaki7.bsky.social
www.arxiv.org/abs/2507.04610
any4: Learned 4-bit Numeric Representation for LLMs
We present any4, a learned 4-bit weight quantization solution for large language models (LLMs) providing arbitrary numeric representations without requiring pre-processing of weights or activations. a...
www.arxiv.org
July 9, 2025 at 4:41 AM
bourbaki7.bsky.social
arxiv.org/abs/2507.02119
Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks
What scaling limits govern neural network training dynamics when model size and training time grow in tandem? We show that despite the complex interactions between architecture, training algorithms, a...
arxiv.org
July 8, 2025 at 6:05 PM
bourbaki7.bsky.social
arxiv.org/abs/2506.20752
Characterization and Mitigation of Training Instabilities in Microscaling Formats
Training large language models is an expensive, compute-bound process that must be repeated as models scale, algorithms improve, and new data is collected. To address this, next-generation hardware ac...
arxiv.org
July 2, 2025 at 8:58 PM
bourbaki7.bsky.social
arxiv.org/abs/2501.11873
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
This paper revisits the implementation of $\textbf{L}$oad-$\textbf{b}$alancing $\textbf{L}$oss (LBL) when training Mixture-of-Experts (MoEs) models. Specifically, LBL for MoEs is defined as $N_E \sum_...
arxiv.org
June 30, 2025 at 6:26 AM
bourbaki7.bsky.social
arxiv.org/abs/2506.04374
A Statistical Physics of Language Model Reasoning
Transformer LMs show emergent reasoning that resists mechanistic understanding. We offer a statistical physics framework for continuous-time chain-of-thought reasoning dynamics. We model sentence-leve...
arxiv.org
June 29, 2025 at 3:26 AM