Existential Risk Observatory
@xrobservatory.bsky.social
Reducing existential risk by informing the public debate. We propose a Conditional AI Safety Treaty: https://time.com/7171432/conditional-ai-safety-treaty-trump/
Xriskers should see the obvious and campaign together with those concerned about data centers, aiming for xrisk awareness raising and getting good regulation implemented.
November 28, 2025 at 11:53 AM
Xriskers should see the obvious and campaign together with those concerned about data centers, aiming for xrisk awareness raising and getting good regulation implemented.
AI using water and energy that was made for human beings is an obvious resource conflict, too. There's a continuum straight from these issues to human replacement and eventually human extinction. The more powerful AI gets, the faster this will go.
November 28, 2025 at 11:53 AM
AI using water and energy that was made for human beings is an obvious resource conflict, too. There's a continuum straight from these issues to human replacement and eventually human extinction. The more powerful AI gets, the faster this will go.
Our core concern is humanity getting replaced by AI. Gradual disempowerment is one scenario many worry about. What failure looks like, where factories start sucking up our oxygen, is another. Even the classic paperclip maximizer scenario is a resource conflict at heart.
November 28, 2025 at 11:53 AM
Our core concern is humanity getting replaced by AI. Gradual disempowerment is one scenario many worry about. What failure looks like, where factories start sucking up our oxygen, is another. Even the classic paperclip maximizer scenario is a resource conflict at heart.
Already, these issues are big enough for politicians from left to right to win elections on. Xriskers can read an exponential curve. If this is true today, imagine what AI politics will look like five years from now!
November 28, 2025 at 11:53 AM
Already, these issues are big enough for politicians from left to right to win elections on. Xriskers can read an exponential curve. If this is true today, imagine what AI politics will look like five years from now!
This trial will be aimed at @stopai.bsky.social, but we all know that Sam Altman is the one doing what should really be illegal.
Congratulations to StopAI for making this happen!
Congratulations to StopAI for making this happen!
November 7, 2025 at 9:59 AM
This trial will be aimed at @stopai.bsky.social, but we all know that Sam Altman is the one doing what should really be illegal.
Congratulations to StopAI for making this happen!
Congratulations to StopAI for making this happen!
Debating this absurd situation in public is badly needed. It's an even better idea to do so with one of the worst perpetrators, who has time and again tried to build exactly the kind of AI that could kill us all, and who has time and again lobbied hard against any regulation aiming to keep us safe.
November 7, 2025 at 9:59 AM
Debating this absurd situation in public is badly needed. It's an even better idea to do so with one of the worst perpetrators, who has time and again tried to build exactly the kind of AI that could kill us all, and who has time and again lobbied hard against any regulation aiming to keep us safe.
Slowly, but surely, the public is getting informed that there is a level of AI that may kill everyone. And obviously, an informed public is not going to let that happen.
Never mind SB1047. In the end, we will win.
Never mind SB1047. In the end, we will win.
June 11, 2025 at 10:13 PM
Slowly, but surely, the public is getting informed that there is a level of AI that may kill everyone. And obviously, an informed public is not going to let that happen.
Never mind SB1047. In the end, we will win.
Never mind SB1047. In the end, we will win.
What is interesting is that the presenter assumes familiarity with not only the possibility that AI could cause our extinction, but also the fact that many experts think there is an appreciable chance this may actually happen.
June 11, 2025 at 10:13 PM
What is interesting is that the presenter assumes familiarity with not only the possibility that AI could cause our extinction, but also the fact that many experts think there is an appreciable chance this may actually happen.
More info and discussion here:
forum.effectivealtruism.org/posts/XJuPEy...
www.lesswrong.com/posts/sc4Kh5...
forum.effectivealtruism.org/posts/XJuPEy...
www.lesswrong.com/posts/sc4Kh5...
March 26, 2025 at 11:50 AM
More info and discussion here:
forum.effectivealtruism.org/posts/XJuPEy...
www.lesswrong.com/posts/sc4Kh5...
forum.effectivealtruism.org/posts/XJuPEy...
www.lesswrong.com/posts/sc4Kh5...
- Offense/defense balance. Many seem to rely on this balance favoring defense, but so far little work has been done on aiming to determine whether this assumption holds, and in fleshing out what such defense could look like. A follow-up research project could be to shed light on these questions.
March 26, 2025 at 11:50 AM
- Offense/defense balance. Many seem to rely on this balance favoring defense, but so far little work has been done on aiming to determine whether this assumption holds, and in fleshing out what such defense could look like. A follow-up research project could be to shed light on these questions.
Our follow-up research might include:
- Systemic risks, such as gradual disempowerment, geopolitical risks (see e.g. MAIM), mass unemployment, stable extreme inequality, planetary boundaries and climate, and others.
- Systemic risks, such as gradual disempowerment, geopolitical risks (see e.g. MAIM), mass unemployment, stable extreme inequality, planetary boundaries and climate, and others.
March 26, 2025 at 11:50 AM
Our follow-up research might include:
- Systemic risks, such as gradual disempowerment, geopolitical risks (see e.g. MAIM), mass unemployment, stable extreme inequality, planetary boundaries and climate, and others.
- Systemic risks, such as gradual disempowerment, geopolitical risks (see e.g. MAIM), mass unemployment, stable extreme inequality, planetary boundaries and climate, and others.
- Require security and governance audits for developers of models above the threshold.
- Impose reporting requirements and Know-Your-Customer requirements on cloud compute providers.
- Verify implementation via oversight of the compute supply chain.
- Impose reporting requirements and Know-Your-Customer requirements on cloud compute providers.
- Verify implementation via oversight of the compute supply chain.
March 26, 2025 at 11:50 AM
- Require security and governance audits for developers of models above the threshold.
- Impose reporting requirements and Know-Your-Customer requirements on cloud compute providers.
- Verify implementation via oversight of the compute supply chain.
- Impose reporting requirements and Know-Your-Customer requirements on cloud compute providers.
- Verify implementation via oversight of the compute supply chain.
Based on our review, our treaty recommendations are:
- Establish a compute threshold above which development should be regulated.
- Require “model audits” (evaluations and red-teaming) for models above the threshold.
- Establish a compute threshold above which development should be regulated.
- Require “model audits” (evaluations and red-teaming) for models above the threshold.
March 26, 2025 at 11:50 AM
Based on our review, our treaty recommendations are:
- Establish a compute threshold above which development should be regulated.
- Require “model audits” (evaluations and red-teaming) for models above the threshold.
- Establish a compute threshold above which development should be regulated.
- Require “model audits” (evaluations and red-teaming) for models above the threshold.
Our paper "International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty" focuses on risk thresholds, types of international agreement, building scientific consensus, standardisation, auditing, verification and incentivisation.
arxiv.org/abs/2503.18956
arxiv.org/abs/2503.18956
International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty
The malicious use or malfunction of advanced general-purpose AI (GPAI) poses risks that, according to leading experts, could lead to the 'marginalisation or extinction of humanity.' To address these r...
arxiv.org
March 26, 2025 at 11:50 AM
Our paper "International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty" focuses on risk thresholds, types of international agreement, building scientific consensus, standardisation, auditing, verification and incentivisation.
arxiv.org/abs/2503.18956
arxiv.org/abs/2503.18956