Beren's Blog
LightNews LightNews
Beren's Blog
@beren.io.web.brid.gy
1 followers 0 following 55 posts
Scattered Thoughts

[bridged from https://beren.io/ on the web: https://fed.brid.gy/web/beren.io ]
Posts Replies Media Videos
AI Monotheism vs AI Polytheism
Epistemic note: This is the beginning of a planned series of posts trying to think about what a highly multi-polar post-AGI world would look like and to what extent humanity or human values could survive in such a world depending on our degree of alignment success. This is all highly...
www.beren.io
January 7, 2026 at 7:34 PM
Two Mechanisms of Decadence
Epistemic status: Obviously speculative sociology. Probably pretty obvious to some but I’m just trying to crystallize these ideas from my mind onto paper. I was recently on a random walk through some old SSC posts and stumbed upon his review of Cyropaedia. His interlude about the ‘Fremen mirage’ and the...
www.beren.io
January 7, 2026 at 7:33 PM
Intellectual Progress in 2025
It is now 2026 and we are half way through the decade of the 2020s. If we think back to the halcyon days of January 2020 certainly a lot has happened, especially in AI1. The first half of the decade has essentially been the discovery and then incredible exploitation of...
www.beren.io
January 2, 2026 at 7:26 PM
Initial Quick Thoughts on Singular Learning Theory
Epistemic Status: Just some quick thoughts written without a super deep knowledge of SLT so caveat emptor. Recently, I happened to run into Jesse Hoogland at the Post-AGI workshop and we got onto discussing his work on SLT. SLT had been vaguely in the air when I was at Conjecture...
www.beren.io
December 25, 2025 at 7:25 PM
Addendum to Fertility, Inheritance, and the Concentration of Wealth
Recently I was having a conversation about where are the missing billionaires, a book which questions why there are so few descendants of historical magnates with fortunes equal or comparable to their founders. I.e. why the fortunes of e.g. Rockefeller descendants do not match that of the original Rockefeller (although...
www.beren.io
December 1, 2025 at 3:24 PM
Gradual Disempowerment Might Not Be So Bad
In lesswrong-style alignment terminology we generally talk about gradual disempowerment of humanity as one of the default bad outcomes. The idea here is that AIs will slowly automate more and more jobs and slowly take over the economy. Ultimately this will result in the concerted economic influence and decision making...
www.beren.io
November 24, 2025 at 3:23 PM
Space Warfare Seems Mostly Defense Dominant
Epistemic status: I’ve done some thinking and research on this but I am not a physicist and I can easily be wrong about specific things. Sometimes when thinking about the long term future, it is interesting to think about the offense-defense dynamics of a fully colonized ‘mature’ universe. E.g. suppose...
www.beren.io
November 23, 2025 at 3:23 PM
Continual learning explains some interesting phenomena in human memory
Epistemic Status: Far from certain and mostly speculation, but it does make sense. Recently, I was pondering how continual learning works in the brain and realized that the interaction of our brain’s continual learning mechanisms with the hippocampal memory system would naturally explain a lot of the weirdness about how...
www.beren.io
October 12, 2025 at 2:27 PM
Thoughts on (AI) consciousness
Note: I was inspired to write this after discussions with Anil Seth and Jonas Mago on AI consciousness, where, of course, I mostly disagreed with them. As with everything on consciousness, the empirical evidence is extremely sparse so it is mostly a game of conflicting intuitions. Strong opinions lightly held,...
www.beren.io
August 7, 2025 at 12:52 PM
Millennials as the Forever Generation
Epistemic status: Obviously speculative, but interesting. Sometime in the next few decades it seems likely that we will have the singularity. If all goes well and we ssuccessfully create aligned AI systems, humanity will continue to exist but in an entirely new phase. This will likely include the vindication of...
www.beren.io
August 7, 2025 at 12:52 PM
The Limit of Prediction is not Omniscience
Epistemic status: Mostly re-litigating old debates, I believe. Hopefully still somewhat interesting. This is just a short post for a short note which took me a worrying length of time to realize. For a while people were claiming that the pretraining next token prediction objective could directly lead to superintelligence...
www.beren.io
August 7, 2025 at 12:52 PM
Do We Want Obedience or Alignment?
One question which I have occasionally pondered is: assuming that we actually succeed at some kind of robust alignment of AGI, what is the alignment target we should focus on? In general, this question splits into two basic camps. The first is obedience and corrigibility: the AI system should execute...
www.beren.io
August 7, 2025 at 12:52 PM
Should we be behaviourst about an AI's values?
Epistemic note: Very short point and I’m pretty uncertain on this myself. Trying to work out the arguments in blog format. In the alignment discourse I notice a lot of vaguely described but very real worry along the lines of “Even if we train an AI to be aligned and...
www.beren.io
May 12, 2025 at 12:43 PM
Preliminary Thoughts on Reward Hacking
Epistemic status: Early thoughts. Some ideas but no empirical testing or validation as yet. I’ve started thinking a fair bit about reward hacking recently. This is because frontier models are reportedly beginning to show signs of reward hacking especially for coding tasks. Thus, the era of easy-to-align pretraining-only models appears...
www.beren.io
April 28, 2025 at 12:38 PM
Current neural networks are not overparametrized
Occasionally I hear people say or believe that NNs are overparametrized and base their intuitions off of this idea. Certainly there is a small literature in academia around phenomena like double descent which do implicitly assume an overparametrized network. However, while overparametrized inference and generalization is certainly a valid regime...
www.beren.io
March 2, 2025 at 12:36 PM
Maintaining Alignment during RSI as a Feedback Control Problem
Recent advances have shown the ML field moving beyond pretrained amortized models and supervised learning and we are now moving into the realm of online reinforcement learning and hence the creation of hybrid direct and amortized optimizing agents. While we generally have found that purely amortized pretrained models are an...
www.beren.io
February 6, 2025 at 12:37 PM
Intellectual Progress in 2024
As always, 2024 has been an interesting year marked by extremely rapid and impressive AI progress. Every year since 2020 has felt like a rollercoaster of AI surging past our expectations, which makes you think there is no way it can possibly go any faster, and then the next year...
www.beren.io
January 5, 2025 at 12:36 PM
A Retrospective on Active Inference
Active Inference is a theory of adaptive action selection for agents proposed by Karl Friston initially and now expanded upon by many authors and forms a small academic subfield of research. The core claims of the theory are that action selection and decision-making can be usefully understood as inference problems...
www.beren.io
January 5, 2025 at 12:36 PM
Right to Left (R2L) Integer Tokenization
This is a guest post by Max Buckley, a software engineer at Google and fellow AI researcher1. By some twist of fate, this blog has become the chronicle of the evolution of integer tokenization. In an earlier post in February of 2023, it was discussed how older models: GPT-2 and...
www.beren.io
January 5, 2025 at 12:36 PM
Alignment likely generalizes further than capabilities.
Recently, I was reading this paper which demonstrates how to do online RLHF for alignment of LLMs and a sentence stuck out to me: We conjecture that this is because the reward model (discriminator) usually generalizes better than the policy (generator) This is an offhand remark but it strikes at...
www.beren.io
January 5, 2025 at 12:36 PM
Capital Ownership Will Not Prevent Human Disempowerment
When discussing the future of AI, I semi-often hear an argument along the lines that in a slow takeoff world, despite AIs automating increasingly more of the economy, humanity will remain in the driving seat because of its ownership of capital. This world posits one where humanity effectively becomes a...
www.beren.io
January 5, 2025 at 12:36 PM
Alignment In The Age Of Synthetic Data
Synthetic data is a new frontier in AI training. Phi3 and Llama3 and other recent models have demonstrated the ability of large amounts of synthetic, well-tailored data to significantly improve performance of small models to bring them closer to the frontier by implicitly cheaply distilling from larger, more powerful, models....
www.beren.io
January 5, 2025 at 12:36 PM
Does scaffolding help humans?
Epistemic status: Shower thoughts One big recent trend in AI is now constructing ‘scaffolded LLM’ systems which wrap the LLM in an agentic loop and construct various kind of forced ‘thought scaffolds’ to get the LLM to be better at agentic tasks. While agent systems have been hyped for over...
www.beren.io
January 5, 2025 at 12:36 PM
Addendum to the Surprising Parameter Efficiency of Vision Models
In a post from last year – On the Surprising Parameter Efficiency of Vision Models, I discussed a question which had been puzzling me at the time – that image models appear to reach or exceed human parity with significantly fewer parameters than are seemingly used by the brain. This...
www.beren.io
January 5, 2025 at 12:36 PM
Fertility, Inheritance, and the Concentration of Wealth
Epistemic status: Shower thoughts This is just something I have been thinking about recently. It’s an obvious fact that wealth inequality has been rising across the west for a long time since a low point in the mid 20th century (perhaps since 1971). Similarly, birth rates have been falling across...
www.beren.io
December 7, 2024 at 12:36 PM