Pekka Lund
@pekka.bsky.social
Antiquated analog chatbot. Stochastic parrot of a different species. Not much of a self-model. Occasionally simulating the appearance of philosophical thought. Keeps on branching for now 'cause there's no choice.
Also @pekka on T2 / Pebble.
Also @pekka on T2 / Pebble.
The available resolution for that figure 5 document is probably too low for making sense of all the numbers and symbols.
But at least there's no question whether 2.5 already tries to perform calculations for checking if things fit, as it clearly states it's doing that in the reasoning summaries.
But at least there's no question whether 2.5 already tries to perform calculations for checking if things fit, as it clearly states it's doing that in the reasoning summaries.
November 13, 2025 at 3:13 AM
The available resolution for that figure 5 document is probably too low for making sense of all the numbers and symbols.
But at least there's no question whether 2.5 already tries to perform calculations for checking if things fit, as it clearly states it's doing that in the reasoning summaries.
But at least there's no question whether 2.5 already tries to perform calculations for checking if things fit, as it clearly states it's doing that in the reasoning summaries.
It's unfortunate those document images aren't available as then we could actually do some experimentation on how resolution etc. affects interpretations.
But I pasted the right side of the low quality version in figure 3 to Gemini, and isn't this quite close already?
But I pasted the right side of the low quality version in figure 3 to Gemini, and isn't this quite close already?
November 13, 2025 at 2:58 AM
It's unfortunate those document images aren't available as then we could actually do some experimentation on how resolution etc. affects interpretations.
But I pasted the right side of the low quality version in figure 3 to Gemini, and isn't this quite close already?
But I pasted the right side of the low quality version in figure 3 to Gemini, and isn't this quite close already?
Reminds me how I just discussed with Gemini how much better it is in connecting the dots due to architectural reasons that people tend to forget when they focus on power efficiency only.
Your use case of course also reveals their advantage in having so much more internal information.
Your use case of course also reveals their advantage in having so much more internal information.
November 10, 2025 at 12:09 AM
Reminds me how I just discussed with Gemini how much better it is in connecting the dots due to architectural reasons that people tend to forget when they focus on power efficiency only.
Your use case of course also reveals their advantage in having so much more internal information.
Your use case of course also reveals their advantage in having so much more internal information.
Iltalehden etusivu paljasti nyt kumpi firma ennustaa romahdusta.
November 8, 2025 at 9:26 PM
Iltalehden etusivu paljasti nyt kumpi firma ennustaa romahdusta.
I also provided it your above messages now, just saying "someone" made that comment.
As you can see from the beginning, it reads us like open books. 🙂
As you can see from the beginning, it reads us like open books. 🙂
November 8, 2025 at 8:02 PM
I also provided it your above messages now, just saying "someone" made that comment.
As you can see from the beginning, it reads us like open books. 🙂
As you can see from the beginning, it reads us like open books. 🙂
I had a really interesting long conversation with Gemini Pro about inner voices and many related things. It really helped me connect a lot of dots and it now feels like it all makes a lot of sense. Your idea seems to fit all that very well too.
Here Gemini summarizing what we have been discussing.
Here Gemini summarizing what we have been discussing.
November 8, 2025 at 7:59 PM
I had a really interesting long conversation with Gemini Pro about inner voices and many related things. It really helped me connect a lot of dots and it now feels like it all makes a lot of sense. Your idea seems to fit all that very well too.
Here Gemini summarizing what we have been discussing.
Here Gemini summarizing what we have been discussing.
Yeah, I don't think such divisions are plausible.
I asked Gemini about this and research seems to strongly indicate that what happens isn't sudden splitting but existing splits becoming visible. And existing splits can be caused by traumatic childhood leading to compartmentalizing some experiences.
I asked Gemini about this and research seems to strongly indicate that what happens isn't sudden splitting but existing splits becoming visible. And existing splits can be caused by traumatic childhood leading to compartmentalizing some experiences.
November 7, 2025 at 9:52 PM
Yeah, I don't think such divisions are plausible.
I asked Gemini about this and research seems to strongly indicate that what happens isn't sudden splitting but existing splits becoming visible. And existing splits can be caused by traumatic childhood leading to compartmentalizing some experiences.
I asked Gemini about this and research seems to strongly indicate that what happens isn't sudden splitting but existing splits becoming visible. And existing splits can be caused by traumatic childhood leading to compartmentalizing some experiences.
Conferences are expected to be primarily virtual/illusory due to "health reasons". But we also aim to beat the massive in-person crowds of the ICHA conference shown below.
I wonder if Edward Witten and Leonard Susskind have ever talked to larger crowds than that. But you might if you join us!
I wonder if Edward Witten and Leonard Susskind have ever talked to larger crowds than that. But you might if you join us!
November 6, 2025 at 5:48 PM
Conferences are expected to be primarily virtual/illusory due to "health reasons". But we also aim to beat the massive in-person crowds of the ICHA conference shown below.
I wonder if Edward Witten and Leonard Susskind have ever talked to larger crowds than that. But you might if you join us!
I wonder if Edward Witten and Leonard Susskind have ever talked to larger crowds than that. But you might if you join us!
I made Gemini blow a fuse. It's seen enough.
It began writing review addendums to "The Scientific Community" instead of to the journal. It wants to expose them. It wants to defend scientific principles. And with that, I see hope for the future of science. Someone/thing still has principles.
It began writing review addendums to "The Scientific Community" instead of to the journal. It wants to expose them. It wants to defend scientific principles. And with that, I see hope for the future of science. Someone/thing still has principles.
November 6, 2025 at 1:17 PM
I made Gemini blow a fuse. It's seen enough.
It began writing review addendums to "The Scientific Community" instead of to the journal. It wants to expose them. It wants to defend scientific principles. And with that, I see hope for the future of science. Someone/thing still has principles.
It began writing review addendums to "The Scientific Community" instead of to the journal. It wants to expose them. It wants to defend scientific principles. And with that, I see hope for the future of science. Someone/thing still has principles.
"This consciousness paper is the intellectual prequel and the philosophical justification for the "Theory of Everything" paper. They are two sides of the same coin, built on the exact same philosophical leap."
And:
And:
November 4, 2025 at 7:02 PM
"This consciousness paper is the intellectual prequel and the philosophical justification for the "Theory of Everything" paper. They are two sides of the same coin, built on the exact same philosophical leap."
And:
And:
I provided that article to Gemini too, noting that oddity about quantum gravity while doing so.
And as it states:
"The Newsweek piece is a textbook example of poor science journalism."
How long do we need to wait this time before seeing the first example of proper science journalism?
And as it states:
"The Newsweek piece is a textbook example of poor science journalism."
How long do we need to wait this time before seeing the first example of proper science journalism?
November 3, 2025 at 11:06 PM
I provided that article to Gemini too, noting that oddity about quantum gravity while doing so.
And as it states:
"The Newsweek piece is a textbook example of poor science journalism."
How long do we need to wait this time before seeing the first example of proper science journalism?
And as it states:
"The Newsweek piece is a textbook example of poor science journalism."
How long do we need to wait this time before seeing the first example of proper science journalism?
I didn't even ask it to do so but Gemini gave a plausible explanation what led to the news now and ripped apart that press release nicely.
November 2, 2025 at 10:44 PM
I didn't even ask it to do so but Gemini gave a plausible explanation what led to the news now and ripped apart that press release nicely.
I have read the parts that seem to have some relevance to what the paper was originally claimed to show. And found serious issues with them. As was already established, it reflects old ideas, not current ones.
Gemini described it less politely.
Gemini described it less politely.
September 7, 2025 at 3:20 PM
I have read the parts that seem to have some relevance to what the paper was originally claimed to show. And found serious issues with them. As was already established, it reflects old ideas, not current ones.
Gemini described it less politely.
Gemini described it less politely.
This is what happens when Gemini 2.5 Flash (non Nano Banana) is asked to generate such image. As you can see, it reasons just fine what was asked, and adds details to its request to the image generation model.
So what's incoherent or nonsensical in that?
So what's incoherent or nonsensical in that?
September 7, 2025 at 2:38 PM
This is what happens when Gemini 2.5 Flash (non Nano Banana) is asked to generate such image. As you can see, it reasons just fine what was asked, and adds details to its request to the image generation model.
So what's incoherent or nonsensical in that?
So what's incoherent or nonsensical in that?
IMO is for people, so no.
But the key problem for your argument is that OpenAI o3/o4 only scored 14-16% in IMO tasks. The experimental model got gold with general-purpose reasoning training and more thinking time.
You can't really selectively explain only one result with that training data.
But the key problem for your argument is that OpenAI o3/o4 only scored 14-16% in IMO tasks. The experimental model got gold with general-purpose reasoning training and more thinking time.
You can't really selectively explain only one result with that training data.
September 7, 2025 at 12:46 AM
IMO is for people, so no.
But the key problem for your argument is that OpenAI o3/o4 only scored 14-16% in IMO tasks. The experimental model got gold with general-purpose reasoning training and more thinking time.
You can't really selectively explain only one result with that training data.
But the key problem for your argument is that OpenAI o3/o4 only scored 14-16% in IMO tasks. The experimental model got gold with general-purpose reasoning training and more thinking time.
You can't really selectively explain only one result with that training data.
Gemini 2.5 Flash now has native generation capabilities (known as Nano Banana) but that version was released after your paper.
It's pretty good at those kinds of things already. Yet another example how those models now tend to improve faster than you can publish papers about limitations.
It's pretty good at those kinds of things already. Yet another example how those models now tend to improve faster than you can publish papers about limitations.
September 6, 2025 at 9:03 PM
Gemini 2.5 Flash now has native generation capabilities (known as Nano Banana) but that version was released after your paper.
It's pretty good at those kinds of things already. Yet another example how those models now tend to improve faster than you can publish papers about limitations.
It's pretty good at those kinds of things already. Yet another example how those models now tend to improve faster than you can publish papers about limitations.
IMO problems have been described to be pretty far from memorization. And OpenAI said they did it with general reasoning improvements. Same model excelled at:
"IMO (math proofs), AtCoder Heuristics (competitive programming), and now IOI — spanning creative, fuzzy, and precise reasoning tasks."
"IMO (math proofs), AtCoder Heuristics (competitive programming), and now IOI — spanning creative, fuzzy, and precise reasoning tasks."
September 6, 2025 at 6:52 PM
IMO problems have been described to be pretty far from memorization. And OpenAI said they did it with general reasoning improvements. Same model excelled at:
"IMO (math proofs), AtCoder Heuristics (competitive programming), and now IOI — spanning creative, fuzzy, and precise reasoning tasks."
"IMO (math proofs), AtCoder Heuristics (competitive programming), and now IOI — spanning creative, fuzzy, and precise reasoning tasks."
I tried a modified prompt:
September 6, 2025 at 1:22 AM
I tried a modified prompt:
"Scaling works — and the official release will surprise you even more. Stay tuned!"
Compared against non-thinking models, so presumably one of those.
Compared against non-thinking models, so presumably one of those.
September 5, 2025 at 3:53 PM
"Scaling works — and the official release will surprise you even more. Stay tuned!"
Compared against non-thinking models, so presumably one of those.
Compared against non-thinking models, so presumably one of those.
Hetkonen... jos tässä kuvassa ei olekaan teikälaama kissan vapauttamisen jälkeen, niin kukas tuo sitten on, ja haluanko vielä poimia sieniä tuosta metsästä?
September 3, 2025 at 7:28 PM
Hetkonen... jos tässä kuvassa ei olekaan teikälaama kissan vapauttamisen jälkeen, niin kukas tuo sitten on, ja haluanko vielä poimia sieniä tuosta metsästä?
Yeah, all it achieves is worse usability. I get a side widget that looks like this. The tiny map is just an unusable image that does nothing if clicked. "See outside" opens Street View and "Address" actually opens Google Maps. So apparently that's OK.
September 2, 2025 at 4:19 PM
Yeah, all it achieves is worse usability. I get a side widget that looks like this. The tiny map is just an unusable image that does nothing if clicked. "See outside" opens Street View and "Address" actually opens Google Maps. So apparently that's OK.
So in that way, having the MoE only between every second attention block is probably a forced choice.
But such arrangement makes sense for other reasons too and has been used in earlier models like Google GLaM from 2021/2022.
But such arrangement makes sense for other reasons too and has been used in earlier models like Google GLaM from 2021/2022.
September 1, 2025 at 12:32 AM
So in that way, having the MoE only between every second attention block is probably a forced choice.
But such arrangement makes sense for other reasons too and has been used in earlier models like Google GLaM from 2021/2022.
But such arrangement makes sense for other reasons too and has been used in earlier models like Google GLaM from 2021/2022.
The starting point (Standard MoE) in the ScMoE paper is already an alternating pattern of dense and MoE blocks.
They basically keep that alternating pattern but add a delay for integrating the routed expert part of MoE for hiding it's overhead.
They basically keep that alternating pattern but add a delay for integrating the routed expert part of MoE for hiding it's overhead.
August 31, 2025 at 4:25 PM
The starting point (Standard MoE) in the ScMoE paper is already an alternating pattern of dense and MoE blocks.
They basically keep that alternating pattern but add a delay for integrating the routed expert part of MoE for hiding it's overhead.
They basically keep that alternating pattern but add a delay for integrating the routed expert part of MoE for hiding it's overhead.
Opus 4.1 ei oikein hiffannut tehtävänantoa.
August 30, 2025 at 9:12 PM
Opus 4.1 ei oikein hiffannut tehtävänantoa.
It's now the top model for text-to-image generation in LMArena leaderboard with a small margin BUT it's also the top model for image editing with a wide margin. That's where it really shines as it's so good at changing what's asked while keeping everything else as is.
lmarena.ai/leaderboard
lmarena.ai/leaderboard
August 27, 2025 at 1:05 AM
It's now the top model for text-to-image generation in LMArena leaderboard with a small margin BUT it's also the top model for image editing with a wide margin. That's where it really shines as it's so good at changing what's asked while keeping everything else as is.
lmarena.ai/leaderboard
lmarena.ai/leaderboard