Email bombing? I was email bombed recently by identity thieves trying to prevent me from noticing the emails pertaining to their fraudulent credit card application in my name. Most of the emails seemed to be from legit mailing lists.

IMO the deeper ground truth is "What works pragmatically to solve collective action problems?" Moral intuitions are adaptive for that, but so is deference to norms, and it's easier to have shared norms if they can be derived from simple principles in a transparent way.

Consistency and legibility are desirable because morality is an evolved mechanism for solving collective action problems. That's easier when allies can assume common knowledge of norms. Relying on opaque, highly context-dependent moral intuitions leaves more hiding places for defection to evolve.

A contrived way to do this would be, take a union of discrete log and SAT, weighted so most instances are discrete log. E.g. say the 2nd half of the input is interpreted as a SAT problem if the 1st half is all 0s, otherwise as discrete log. Unenlightening, but it's at least an existence proof.

... "We have to get a back-up planet set up because killer asteroids," as if that couldn't wait until after we have the tech for Dyson spheres or whatever a more advanced civilization would do with our solar system.

Or maybe not it's not entirely the people involved consciously wanting prestige, but evolution itself, which rewards us for exploration and prestigious achievement with feelings of wonder and romance, but also causes us to confabulate irrational motives for doing those things, like ...

Yes. I haven't seen any sensible justification for e.g. a manned Mars mission with pre-Singularity tech. I think people just want the prestige of being one of the first n people on Mars for small n, or of contributing to that achievement in some way. Or they want to admire people who do that.

For all that, I still put *some* nontrivial credence on another AI winter, but I'm baffled that so many people predict it with confidence. Why couldn't the current trend just keep going until it blows right past human-level AGI?

People with inside knowledge of the frontier labs, like Ilya Sutskever, Leopold Aschenbrenner, and Christian Szegedy, keep telling us we'll have further amazing progress in the next few years. These seem like serious people who wouldn't just bullshit us about that.
x.com/ChrSzegedy/s...

Progress on reasoning seems good, judging by recent work on coding agents, the Reuters article above, and Deepmind's AlphaProof IMO results. Progress in robotics seems good and will be a source of new training data.

Why tho? What's the obstacle that's going to stop the rapid progress we've seen over the past decade? AFAICT there's still plenty of room to spend more and use better hardware for bigger training runs. Maybe we're approaching a training data wall, but I doubt we're optimally using the data we have …

I want more people to take seriously the possibility that we will soon have AIs that are useful in *exactly* the ways human artists and scientists are useful, and much, much more besides. I find the implications staggering and terrifying.

www.reuters.com/technology/a...