Nicholas Guttenberg
@ngutten.bsky.social
Hm, I'm noticing that when injecting arbitrary vectors into Qwen3's input (in place of the actual learned embeddings) they seem to be interpreted very strongly as short sequences of letters, not as something more abstract.
Is there some way to quantify the 'concreteness' of an input layer now?
Is there some way to quantify the 'concreteness' of an input layer now?
November 4, 2025 at 1:39 AM
Hm, I'm noticing that when injecting arbitrary vectors into Qwen3's input (in place of the actual learned embeddings) they seem to be interpreted very strongly as short sequences of letters, not as something more abstract.
Is there some way to quantify the 'concreteness' of an input layer now?
Is there some way to quantify the 'concreteness' of an input layer now?
Reposted by Nicholas Guttenberg
Dilemmas allow us to deploy conflict, or support, they allow galaxy sized adventure, or quiet literary novels.
What are the tough choices the character faces that are tough *for them* gives me more tools.
Conflict and flaws are almost orthodoxy, it also cuts off a wide amount of material for me.
What are the tough choices the character faces that are tough *for them* gives me more tools.
Conflict and flaws are almost orthodoxy, it also cuts off a wide amount of material for me.
October 13, 2025 at 4:43 PM
Dilemmas allow us to deploy conflict, or support, they allow galaxy sized adventure, or quiet literary novels.
What are the tough choices the character faces that are tough *for them* gives me more tools.
Conflict and flaws are almost orthodoxy, it also cuts off a wide amount of material for me.
What are the tough choices the character faces that are tough *for them* gives me more tools.
Conflict and flaws are almost orthodoxy, it also cuts off a wide amount of material for me.
I'm perpetually annoyed at how hard it is to do really simple but 'weird' stuff with the transformers and trl libraries. Like, if I want to replicate DeepSeek training, sure you can do that! But now I want the input to be a sequence of arbitrary vectors rather text? Time to rewrite GRPO from scratch
August 28, 2025 at 12:28 AM
I'm perpetually annoyed at how hard it is to do really simple but 'weird' stuff with the transformers and trl libraries. Like, if I want to replicate DeepSeek training, sure you can do that! But now I want the input to be a sequence of arbitrary vectors rather text? Time to rewrite GRPO from scratch
Reposted by Nicholas Guttenberg
Blogpost to read today: strong argument that excessive focus on the first tokens is not something learned from data distribution (like model should naturally "care" about the start of the text to grasp the rest) but a fundamental feature of attention graph. publish.obsidian.md/the-tensor-t...
August 24, 2025 at 4:33 PM
Blogpost to read today: strong argument that excessive focus on the first tokens is not something learned from data distribution (like model should naturally "care" about the start of the text to grasp the rest) but a fundamental feature of attention graph. publish.obsidian.md/the-tensor-t...
I know there's been stuff testing metacognition in various neural networks including LLMs, but is there anything (e.g. post chain-of-thought) about specifically *teaching* LLMs to report metacognitive information?
August 21, 2025 at 7:21 PM
I know there's been stuff testing metacognition in various neural networks including LLMs, but is there anything (e.g. post chain-of-thought) about specifically *teaching* LLMs to report metacognitive information?
Reposted by Nicholas Guttenberg
I think I would have agreed with this more when I was younger. this is probably ~how I felt when I was graduating high school, and then later design school, and into my early career
and then I just started doing off-meta builds and those weird random things kept happening and now I believe in that
and then I just started doing off-meta builds and those weird random things kept happening and now I believe in that
even if this isn't quite true now, it does seem like that's the direction we're headed in
July 29, 2025 at 12:42 PM
I think I would have agreed with this more when I was younger. this is probably ~how I felt when I was graduating high school, and then later design school, and into my early career
and then I just started doing off-meta builds and those weird random things kept happening and now I believe in that
and then I just started doing off-meta builds and those weird random things kept happening and now I believe in that
Promising! I take the sample, apply voltage Vh, let go, then measure the voltage Vc and take dVc/dVh. The negative portion is an interesting feature - I think its because of alterations to the electrodes.
Now if I only had a thousand or so soil samples with known nutrient profiles to train on...
Now if I only had a thousand or so soil samples with known nutrient profiles to train on...
June 28, 2025 at 9:09 PM
Promising! I take the sample, apply voltage Vh, let go, then measure the voltage Vc and take dVc/dVh. The negative portion is an interesting feature - I think its because of alterations to the electrodes.
Now if I only had a thousand or so soil samples with known nutrient profiles to train on...
Now if I only had a thousand or so soil samples with known nutrient profiles to train on...
Doing some experimentation with charging/discharging curves of flooded soils. There's a weird voltage-dependent effect I don't understand that seems sensitive to the soil type. Or of course there could be a bug. Specifically, this immediate drop.
June 22, 2025 at 11:23 PM
Doing some experimentation with charging/discharging curves of flooded soils. There's a weird voltage-dependent effect I don't understand that seems sensitive to the soil type. Or of course there could be a bug. Specifically, this immediate drop.
If we have some machine learning model that is trying to make a prediction (class, next token, whatever) based on different potential redundant sources of evidence (including its own learned priors) is there some way to predict which evidence channels will be favored? To influence that?
June 21, 2025 at 4:52 PM
If we have some machine learning model that is trying to make a prediction (class, next token, whatever) based on different potential redundant sources of evidence (including its own learned priors) is there some way to predict which evidence channels will be favored? To influence that?
My greenhouse sensor project is making me think that maybe those green covers (vs the clear covers) are a good idea for me. Heat management now seems like the biggest issue, and I'm finding temperature is more strongly driven by the last 20 minutes of light than by changes to ventilation.
June 1, 2025 at 6:37 PM
My greenhouse sensor project is making me think that maybe those green covers (vs the clear covers) are a good idea for me. Heat management now seems like the biggest issue, and I'm finding temperature is more strongly driven by the last 20 minutes of light than by changes to ventilation.
First time I've seen a practical application of fractal calculus! Or whatever it is you call fractional order integrals/derivatives... en.wikipedia.org/wiki/Neopola...
Neopolarogram - Wikipedia
en.wikipedia.org
May 31, 2025 at 1:46 AM
First time I've seen a practical application of fractal calculus! Or whatever it is you call fractional order integrals/derivatives... en.wikipedia.org/wiki/Neopola...
Messed around with making a digital theremin from a capacitative sensing board. Drifting calibration makes this really painful and MIDI seems bad for smooth pitch shifts and volume envelopes. I still think the potential is there, but I need to do it another way.
May 29, 2025 at 9:48 PM
Messed around with making a digital theremin from a capacitative sensing board. Drifting calibration makes this really painful and MIDI seems bad for smooth pitch shifts and volume envelopes. I still think the potential is there, but I need to do it another way.
Reposted by Nicholas Guttenberg
On better vs worse directions for using LLMs to simulate behavioral participants in social science research statmodeling.stat.columbia.edu/2025/05/29/l...
LLMs as behavioral study participants | Statistical Modeling, Causal Inference, and Social Science
statmodeling.stat.columbia.edu
May 29, 2025 at 6:19 PM
On better vs worse directions for using LLMs to simulate behavioral participants in social science research statmodeling.stat.columbia.edu/2025/05/29/l...
Today's light curve in the greenhouse. What the heck is that jump near the end there?!
May 28, 2025 at 3:20 AM
Today's light curve in the greenhouse. What the heck is that jump near the end there?!
Is there something like a framework in which its clear when you can and cannot generate data, and what that data can be 'about'?
If I want to infer free floating signals from each-other I just need to be able to synchronize those signals. But if I want to connect something to a ground truth...
If I want to infer free floating signals from each-other I just need to be able to synchronize those signals. But if I want to connect something to a ground truth...
May 28, 2025 at 1:57 AM
Is there something like a framework in which its clear when you can and cannot generate data, and what that data can be 'about'?
If I want to infer free floating signals from each-other I just need to be able to synchronize those signals. But if I want to connect something to a ground truth...
If I want to infer free floating signals from each-other I just need to be able to synchronize those signals. But if I want to connect something to a ground truth...
Well, I got a bunch of sensors and took data once a minute in my greenhouse. It gets up to 110F in there during the day! It's mostly driven by the light - when a cloud passes, the temperature drops within a minute or two.
The color of the light does some interesting things too...
The color of the light does some interesting things too...
May 27, 2025 at 3:58 AM
Well, I got a bunch of sensors and took data once a minute in my greenhouse. It gets up to 110F in there during the day! It's mostly driven by the light - when a cloud passes, the temperature drops within a minute or two.
The color of the light does some interesting things too...
The color of the light does some interesting things too...
Found samphire at a farmer's market. My understanding was, this is the same as sea beans or salicornia, but at least at this stage it tastes and looks pretty different. Are there significantly different varietals between e.g. the UK variant and what would grow in PNW marshes?
May 17, 2025 at 11:30 PM
Found samphire at a farmer's market. My understanding was, this is the same as sea beans or salicornia, but at least at this stage it tastes and looks pretty different. Are there significantly different varietals between e.g. the UK variant and what would grow in PNW marshes?
Crude impedance spectroscopy with a Raspberry Pi Pico. Really crude. Drive the sample with square waves at different frequencies via PWM, Fourier transform, divide, and hope. Not supposed to put a capacitative load on GPIO pins? Uh, well nothing happened, its fine right?
May 13, 2025 at 4:15 AM
Crude impedance spectroscopy with a Raspberry Pi Pico. Really crude. Drive the sample with square waves at different frequencies via PWM, Fourier transform, divide, and hope. Not supposed to put a capacitative load on GPIO pins? Uh, well nothing happened, its fine right?
Reposted by Nicholas Guttenberg
Multi-modal Chain-of-Thought reasoning is already a big thing. That is, CoT models can "reason" not only with chains of text self-prompting, but also by producing images, audio, and more, and engaging with the stimuli they self-produce in a variety of ways.
arxiv.org/abs/2503.12605
arxiv.org/abs/2503.12605
May 9, 2025 at 3:38 PM
Multi-modal Chain-of-Thought reasoning is already a big thing. That is, CoT models can "reason" not only with chains of text self-prompting, but also by producing images, audio, and more, and engaging with the stimuli they self-produce in a variety of ways.
arxiv.org/abs/2503.12605
arxiv.org/abs/2503.12605
Since I've been messing around with a Pi Pico, I've noticed something about working with hobby-scale electronics. It feels like there's always some little part you wish you had, that you forgot to order. Oh if only I had a DAC I could... oh, an opamp would make this better... 1/2
May 7, 2025 at 4:05 AM
Since I've been messing around with a Pi Pico, I've noticed something about working with hobby-scale electronics. It feels like there's always some little part you wish you had, that you forgot to order. Oh if only I had a DAC I could... oh, an opamp would make this better... 1/2
So I'm starting to learn to mess around with a Pi Pico. In theory it can sample an input at 500kHz though it doesn't have the memory to store even a second of that. What sorts of things might be interesting to measure at that rate?
May 4, 2025 at 3:57 AM
So I'm starting to learn to mess around with a Pi Pico. In theory it can sample an input at 500kHz though it doesn't have the memory to store even a second of that. What sorts of things might be interesting to measure at that rate?
Reposted by Nicholas Guttenberg
I'm organizing a "Data for Good Rapid Response Team", i.e. a set of people who can be mobilized to work on short data science projects to help various groups and organizations. Sign up here if this is of interest to you, and please share with others with data science skills.
Data-for-good rapid response team
I'm collecting people with data science skills who may want to hear about sporadic opportunities to use their skills for good on short projects. Project themes will center on topics I am connected wit...
docs.google.com
March 23, 2025 at 5:21 PM
I'm organizing a "Data for Good Rapid Response Team", i.e. a set of people who can be mobilized to work on short data science projects to help various groups and organizations. Sign up here if this is of interest to you, and please share with others with data science skills.
What's the state of the art in causal forest fire spread prediction? Something where you could put in a topographical map and satellite photo and say 'if we made a firebreak here, it would maximally reduce expected fire sizes in the area'.
March 23, 2025 at 4:10 AM
What's the state of the art in causal forest fire spread prediction? Something where you could put in a topographical map and satellite photo and say 'if we made a firebreak here, it would maximally reduce expected fire sizes in the area'.
Reposted by Nicholas Guttenberg
🧪⚛️
I'll be talking at the #apsMarchMeeting 3:36 pm – 4:12 pm this afternoon on "How does non-living matter exhibit open-ended evolution?" Since all life is made up of non-living matter & thus emergent.
A synthesis of earlier work with @ngutten.bsky.social + recent.
summit.aps.org/events/MAR-J...
I'll be talking at the #apsMarchMeeting 3:36 pm – 4:12 pm this afternoon on "How does non-living matter exhibit open-ended evolution?" Since all life is made up of non-living matter & thus emergent.
A synthesis of earlier work with @ngutten.bsky.social + recent.
summit.aps.org/events/MAR-J...
Signatures of Life in Non-Living matter
3:00 pm – 6:00 pm, Tuesday March 18, Session MAR-J56, Anaheim Hilton, California C (Ballroom Level)
summit.aps.org
March 18, 2025 at 1:54 PM
🧪⚛️
I'll be talking at the #apsMarchMeeting 3:36 pm – 4:12 pm this afternoon on "How does non-living matter exhibit open-ended evolution?" Since all life is made up of non-living matter & thus emergent.
A synthesis of earlier work with @ngutten.bsky.social + recent.
summit.aps.org/events/MAR-J...
I'll be talking at the #apsMarchMeeting 3:36 pm – 4:12 pm this afternoon on "How does non-living matter exhibit open-ended evolution?" Since all life is made up of non-living matter & thus emergent.
A synthesis of earlier work with @ngutten.bsky.social + recent.
summit.aps.org/events/MAR-J...
Added a bit of vital wheat gluten and the dough was strong enough I could basically stretch it very thin by hand (and then deepfry it). Its more chewy than crispy in the end, but good.
March 3, 2025 at 2:02 AM
Added a bit of vital wheat gluten and the dough was strong enough I could basically stretch it very thin by hand (and then deepfry it). Its more chewy than crispy in the end, but good.