Anton Leicht
@antonleicht.bsky.social
Politics & Frontier AI Policy | PhD Candidate AI Agent Alignment
antonleicht.me
antonleicht.me
Find the full piece here: www.antonleicht.me/writing/evals (12/12)
Problems in the AI Eval Political Economy — Anton Leicht
Evaluations of new AI models’ capabilities and risks are an important cornerstone of safety-focused AI policy. Currently, their future as part of the policy platform faces peril for four reasons: Entanglements with a broad AI safety ecosystem, structural incentives favouring less helpful evals, susc
www.antonleicht.me
December 17, 2024 at 12:00 PM
Find the full piece here: www.antonleicht.me/writing/evals (12/12)
So what to do? From the political POV, I think:
• More separation between eval technical work and policy advocacy
• Focus on propensity over capability
• Clearer articulation of implications for deployment
• Greater focus on frameworks, fewer evals for evals' sake
(11/N)
• More separation between eval technical work and policy advocacy
• Focus on propensity over capability
• Clearer articulation of implications for deployment
• Greater focus on frameworks, fewer evals for evals' sake
(11/N)
December 17, 2024 at 12:00 PM
So what to do? From the political POV, I think:
• More separation between eval technical work and policy advocacy
• Focus on propensity over capability
• Clearer articulation of implications for deployment
• Greater focus on frameworks, fewer evals for evals' sake
(11/N)
• More separation between eval technical work and policy advocacy
• Focus on propensity over capability
• Clearer articulation of implications for deployment
• Greater focus on frameworks, fewer evals for evals' sake
(11/N)
And if current evals had implications for deployment (I don't think so), this should be made clear, and they should not be used for developer PR and frog-boiling.
As long as concerning evals lead to getting used to inaction, more evals are not as politically helpful. (10/N)
As long as concerning evals lead to getting used to inaction, more evals are not as politically helpful. (10/N)
December 17, 2024 at 12:00 PM
And if current evals had implications for deployment (I don't think so), this should be made clear, and they should not be used for developer PR and frog-boiling.
As long as concerning evals lead to getting used to inaction, more evals are not as politically helpful. (10/N)
As long as concerning evals lead to getting used to inaction, more evals are not as politically helpful. (10/N)
4️⃣Getting used to inaction: Developers keep releasing models with 'concerning' evals. This wears down the future impact of eval orgs expressing concern.
If current evals don't warrant deployment delays, then it should be made clear that only the trajectory is concerning. (9/N)
If current evals don't warrant deployment delays, then it should be made clear that only the trajectory is concerning. (9/N)
December 17, 2024 at 12:00 PM
4️⃣Getting used to inaction: Developers keep releasing models with 'concerning' evals. This wears down the future impact of eval orgs expressing concern.
If current evals don't warrant deployment delays, then it should be made clear that only the trajectory is concerning. (9/N)
If current evals don't warrant deployment delays, then it should be made clear that only the trajectory is concerning. (9/N)
As one exhibit, see Apollo's CEO Marius Hobbhahn defending results against both ducks and rabbits simultaenously: (8/N)
December 17, 2024 at 12:00 PM
As one exhibit, see Apollo's CEO Marius Hobbhahn defending results against both ducks and rabbits simultaenously: (8/N)
3️⃣ The 'rabbit-duck-problem'. Exactly the same eval results simultaneously allow for catastrophization and dismissal.
Is this evidence for impending doom, or evidence that someone asked a robot to pretend to be evil? Turns out, both. (7/N)
Is this evidence for impending doom, or evidence that someone asked a robot to pretend to be evil? Turns out, both. (7/N)
December 17, 2024 at 12:00 PM
3️⃣ The 'rabbit-duck-problem'. Exactly the same eval results simultaneously allow for catastrophization and dismissal.
Is this evidence for impending doom, or evidence that someone asked a robot to pretend to be evil? Turns out, both. (7/N)
Is this evidence for impending doom, or evidence that someone asked a robot to pretend to be evil? Turns out, both. (7/N)
@sebk.bsky.social puts the source of the resulting confusion very succinctly: (6/N)
December 17, 2024 at 12:00 PM
@sebk.bsky.social puts the source of the resulting confusion very succinctly: (6/N)
2️⃣ Incentive structures favor dramatic capability findings over propensity assessments. Eval orgs, media, and developers are all incentivized to amplify 'scary' capability results. It makes for good, but confusing, headlines. (5/N)
December 17, 2024 at 12:00 PM
2️⃣ Incentive structures favor dramatic capability findings over propensity assessments. Eval orgs, media, and developers are all incentivized to amplify 'scary' capability results. It makes for good, but confusing, headlines. (5/N)
1️⃣ Entanglements. Many eval orgs are funded by the same sources that fund policy orgs pushing for evals. This is easily exploitable by opponents.
This already started to show in the SB-1047 discussion: (4/N)
This already started to show in the SB-1047 discussion: (4/N)
December 17, 2024 at 12:00 PM
1️⃣ Entanglements. Many eval orgs are funded by the same sources that fund policy orgs pushing for evals. This is easily exploitable by opponents.
This already started to show in the SB-1047 discussion: (4/N)
This already started to show in the SB-1047 discussion: (4/N)
Down the road, evals need two kinds of policy support: Integration as policy instruments to operationalize restrictions, and ecosystem support. Right now, they're getting some of the latter, but headwinds are building. (3/N)
December 17, 2024 at 12:00 PM
Down the road, evals need two kinds of policy support: Integration as policy instruments to operationalize restrictions, and ecosystem support. Right now, they're getting some of the latter, but headwinds are building. (3/N)
Find the full post here: (2/N)
antonleicht.me/writing/evals
antonleicht.me/writing/evals
Problems in the AI Eval Political Economy — Anton Leicht
Evaluations of new AI models’ capabilities and risks are an important cornerstone of safety-focused AI policy. Currently, their future as part of the policy platform faces peril for four reasons: Entanglements with a broad AI safety ecosystem, structural incentives favouring less helpful evals, susc
www.antonleicht.me
December 17, 2024 at 12:00 PM
Find the full post here: (2/N)
antonleicht.me/writing/evals
antonleicht.me/writing/evals
This strategy is well-precedented. In the past, threats of delays have often been rolled back again fairly quickly, like in numerous conflicts between the EU and Meta or Apple.
It's a little disheartening to see usually cooler heads and elsewhere jump on the bait. (10/N)
It's a little disheartening to see usually cooler heads and elsewhere jump on the bait. (10/N)
December 10, 2024 at 6:42 PM
This strategy is well-precedented. In the past, threats of delays have often been rolled back again fairly quickly, like in numerous conflicts between the EU and Meta or Apple.
It's a little disheartening to see usually cooler heads and elsewhere jump on the bait. (10/N)
It's a little disheartening to see usually cooler heads and elsewhere jump on the bait. (10/N)
Second, it creates precedent.
Lots of jurisdictions are considering AI reg. at the moment. Delays foster the impression that regulating AI leads to frustration and constraints on business.
That's fair - but it means OpenAI likely isn't only focused on making Sora available in Europe ASAP. (9/N)
Lots of jurisdictions are considering AI reg. at the moment. Delays foster the impression that regulating AI leads to frustration and constraints on business.
That's fair - but it means OpenAI likely isn't only focused on making Sora available in Europe ASAP. (9/N)
December 10, 2024 at 6:42 PM
Second, it creates precedent.
Lots of jurisdictions are considering AI reg. at the moment. Delays foster the impression that regulating AI leads to frustration and constraints on business.
That's fair - but it means OpenAI likely isn't only focused on making Sora available in Europe ASAP. (9/N)
Lots of jurisdictions are considering AI reg. at the moment. Delays foster the impression that regulating AI leads to frustration and constraints on business.
That's fair - but it means OpenAI likely isn't only focused on making Sora available in Europe ASAP. (9/N)
Delaying Sora is good lobbying strategy:
First, it focuses latent frustration on uncomfortable policy, e.g. on AIA implementation through the Code of Practice and nat. laws.
Through delays, OpenAI is leveraging public & business pressure to weaken this implementation. (8/N)
First, it focuses latent frustration on uncomfortable policy, e.g. on AIA implementation through the Code of Practice and nat. laws.
Through delays, OpenAI is leveraging public & business pressure to weaken this implementation. (8/N)
December 10, 2024 at 6:42 PM
Delaying Sora is good lobbying strategy:
First, it focuses latent frustration on uncomfortable policy, e.g. on AIA implementation through the Code of Practice and nat. laws.
Through delays, OpenAI is leveraging public & business pressure to weaken this implementation. (8/N)
First, it focuses latent frustration on uncomfortable policy, e.g. on AIA implementation through the Code of Practice and nat. laws.
Through delays, OpenAI is leveraging public & business pressure to weaken this implementation. (8/N)
‘Sora's delay is bad because it means some models won't be available in Europe at all'
I just don't think there is good evidence for that takeaway. If major models actually skip Europe, I'll be very concerned.
But currently, statements like
the below have better alternative explanations. (7/N)
I just don't think there is good evidence for that takeaway. If major models actually skip Europe, I'll be very concerned.
But currently, statements like
the below have better alternative explanations. (7/N)
December 10, 2024 at 6:42 PM
‘Sora's delay is bad because it means some models won't be available in Europe at all'
I just don't think there is good evidence for that takeaway. If major models actually skip Europe, I'll be very concerned.
But currently, statements like
the below have better alternative explanations. (7/N)
I just don't think there is good evidence for that takeaway. If major models actually skip Europe, I'll be very concerned.
But currently, statements like
the below have better alternative explanations. (7/N)
I think that's because o1 competes in a tight language model market, and OpenAI can't afford surrendering first-mover advantage to advanced reasoning in a big market.
As competition around new model classes heats up, that effect will carry over - and delays will become rarer. (6/N)
As competition around new model classes heats up, that effect will carry over - and delays will become rarer. (6/N)
December 10, 2024 at 6:42 PM
I think that's because o1 competes in a tight language model market, and OpenAI can't afford surrendering first-mover advantage to advanced reasoning in a big market.
As competition around new model classes heats up, that effect will carry over - and delays will become rarer. (6/N)
As competition around new model classes heats up, that effect will carry over - and delays will become rarer. (6/N)
'Sora's delay is bad because it means more future delays'
Compounding costs from further delays are very concerning. I think they're a little less likely than reported: OpenAI has delayed Sora, but not o1, a similarly regulation-relevant model (s. the scorecard). (5.5/N)
Compounding costs from further delays are very concerning. I think they're a little less likely than reported: OpenAI has delayed Sora, but not o1, a similarly regulation-relevant model (s. the scorecard). (5.5/N)
December 10, 2024 at 6:42 PM
'Sora's delay is bad because it means more future delays'
Compounding costs from further delays are very concerning. I think they're a little less likely than reported: OpenAI has delayed Sora, but not o1, a similarly regulation-relevant model (s. the scorecard). (5.5/N)
Compounding costs from further delays are very concerning. I think they're a little less likely than reported: OpenAI has delayed Sora, but not o1, a similarly regulation-relevant model (s. the scorecard). (5.5/N)
It's a delay that might be costly to some runaway early adoption use-cases, but runaway early adoption is usually not the European path to profitability anyways. (5/N)
December 10, 2024 at 6:42 PM
It's a delay that might be costly to some runaway early adoption use-cases, but runaway early adoption is usually not the European path to profitability anyways. (5/N)
'Sora's delay is bad because the EU economy needs Sora now'
This seems unlikely to matter at the scale that the doom-saying implies. The first couple months of OpenAI releases often see growing pains anyways, and economic integration of video generation will take some time. (4.5/N)
This seems unlikely to matter at the scale that the doom-saying implies. The first couple months of OpenAI releases often see growing pains anyways, and economic integration of video generation will take some time. (4.5/N)
December 10, 2024 at 6:42 PM
'Sora's delay is bad because the EU economy needs Sora now'
This seems unlikely to matter at the scale that the doom-saying implies. The first couple months of OpenAI releases often see growing pains anyways, and economic integration of video generation will take some time. (4.5/N)
This seems unlikely to matter at the scale that the doom-saying implies. The first couple months of OpenAI releases often see growing pains anyways, and economic integration of video generation will take some time. (4.5/N)
But assuming the regulatory landscape will stay as it is, how could the delay of Sora mean that 'Europe is so fucked' - why is it bad for the EU economy?
Three options: Sora delay is costly, future delays will be costly, future skipped releases will be costly. (4/N)
Three options: Sora delay is costly, future delays will be costly, future skipped releases will be costly. (4/N)
December 10, 2024 at 6:42 PM
But assuming the regulatory landscape will stay as it is, how could the delay of Sora mean that 'Europe is so fucked' - why is it bad for the EU economy?
Three options: Sora delay is costly, future delays will be costly, future skipped releases will be costly. (4/N)
Three options: Sora delay is costly, future delays will be costly, future skipped releases will be costly. (4/N)
Delays follow from an asymmetry: UK and EU are regulated, many other markets aren't.
But that could change: Many other markets (e.g. US states - map below from June 2024) might also regulate soon. Once they reach critical mass, local delays wouldn't be realistic anymore. (3/N)
But that could change: Many other markets (e.g. US states - map below from June 2024) might also regulate soon. Once they reach critical mass, local delays wouldn't be realistic anymore. (3/N)
December 10, 2024 at 6:42 PM
Delays follow from an asymmetry: UK and EU are regulated, many other markets aren't.
But that could change: Many other markets (e.g. US states - map below from June 2024) might also regulate soon. Once they reach critical mass, local delays wouldn't be realistic anymore. (3/N)
But that could change: Many other markets (e.g. US states - map below from June 2024) might also regulate soon. Once they reach critical mass, local delays wouldn't be realistic anymore. (3/N)
To some, this might not be bad news: The delay means that OpenAI is compelled by EU consumer market power to make Sora more compliant (≈safer?)
But Europe falling on its sword to make US AI safer wouldn't be sound political strategy. I instead focus on the economic side: (2/N)
But Europe falling on its sword to make US AI safer wouldn't be sound political strategy. I instead focus on the economic side: (2/N)
December 10, 2024 at 6:42 PM
To some, this might not be bad news: The delay means that OpenAI is compelled by EU consumer market power to make Sora more compliant (≈safer?)
But Europe falling on its sword to make US AI safer wouldn't be sound political strategy. I instead focus on the economic side: (2/N)
But Europe falling on its sword to make US AI safer wouldn't be sound political strategy. I instead focus on the economic side: (2/N)
Why the delay?
No official info, but it's unlikely that this is (only) about the EU AI Act: It's also delayed in the UK, which has no AIA. More likely to be about the ~similar EU Digital Markets Act and UK Online Safety Act.
The Guardian has a similar interpretation: (1/N)
No official info, but it's unlikely that this is (only) about the EU AI Act: It's also delayed in the UK, which has no AIA. More likely to be about the ~similar EU Digital Markets Act and UK Online Safety Act.
The Guardian has a similar interpretation: (1/N)
December 10, 2024 at 6:42 PM
Why the delay?
No official info, but it's unlikely that this is (only) about the EU AI Act: It's also delayed in the UK, which has no AIA. More likely to be about the ~similar EU Digital Markets Act and UK Online Safety Act.
The Guardian has a similar interpretation: (1/N)
No official info, but it's unlikely that this is (only) about the EU AI Act: It's also delayed in the UK, which has no AIA. More likely to be about the ~similar EU Digital Markets Act and UK Online Safety Act.
The Guardian has a similar interpretation: (1/N)
A ray of hope: Institutions like the AI Safety Institutes are genuinely effective and can be politically valuable down the road.
But as they get associated with unpopular policy, they’re at risk, too - like with Sen Cruz' attacks on NIST. (5/N)
But as they get associated with unpopular policy, they’re at risk, too - like with Sen Cruz' attacks on NIST. (5/N)
December 8, 2024 at 8:58 AM
A ray of hope: Institutions like the AI Safety Institutes are genuinely effective and can be politically valuable down the road.
But as they get associated with unpopular policy, they’re at risk, too - like with Sen Cruz' attacks on NIST. (5/N)
But as they get associated with unpopular policy, they’re at risk, too - like with Sen Cruz' attacks on NIST. (5/N)