RL and the Brain seminar @ HMS
@rlbrainseminar.bsky.social
RL and the Brain seminar at Harvard Medical School. Every other Thursday in WAB 236, 12:15 PM - 1:30 PM. See http://rlandthebrain.com for more details.
For more, see Jonathan's talk, or his recently published Nature paper "Natural behaviour is learned through dopamine-mediated reinforcement". Link: www.nature.com/articles/s41...
Natural behaviour is learned through dopamine-mediated reinforcement - Nature
Studies in zebra finches show that dopamine has a key role as a reinforcement signal in the trial-and-error process of learning that underlies complex natural behaviours.
www.nature.com
May 8, 2025 at 2:21 PM
For more, see Jonathan's talk, or his recently published Nature paper "Natural behaviour is learned through dopamine-mediated reinforcement". Link: www.nature.com/articles/s41...
Among other reasons, this kind of work is interesting because it gives us a sense of what vocal acquisition in other animals (like humans!) could be like. It isn't critical for there to be an explicit external reward in order to learn using an actor-critic-like architecture.
May 8, 2025 at 2:21 PM
Among other reasons, this kind of work is interesting because it gives us a sense of what vocal acquisition in other animals (like humans!) could be like. It isn't critical for there to be an explicit external reward in order to learn using an actor-critic-like architecture.
Jonathan et al.'s work shows that song learning in juvenile zebra finches works like that. Dopamine activity is higher than baseline (things 'better than expected') if a song performance more closely resembles the final song, and lower than baseline if it was more dissimilar.
May 8, 2025 at 2:21 PM
Jonathan et al.'s work shows that song learning in juvenile zebra finches works like that. Dopamine activity is higher than baseline (things 'better than expected') if a song performance more closely resembles the final song, and lower than baseline if it was more dissimilar.
Each time you try to sing a song, even if you're not provided explicit feedback, you can provide *yourself* with feedback based on your own internal model of how the song should sound. Did the first part sound right? Was the pitch too high? Internal feedback is a learning signal!
May 8, 2025 at 2:21 PM
Each time you try to sing a song, even if you're not provided explicit feedback, you can provide *yourself* with feedback based on your own internal model of how the song should sound. Did the first part sound right? Was the pitch too high? Internal feedback is a learning signal!
Dopamine is thought to be play an important causal role in trial-and-error learning, especially when it comes to behaviors reinforced by rewards. But what about behaviors without explicit rewards, like learning to sing a song well?
May 8, 2025 at 2:21 PM
Dopamine is thought to be play an important causal role in trial-and-error learning, especially when it comes to behaviors reinforced by rewards. But what about behaviors without explicit rewards, like learning to sing a song well?
For more, see Sonja's talk, or this recent article: news.harvard.edu/gazette/stor...
What electric fish can teach scientists about NeuroAI — Harvard Gazette
Modeling their behaviors may help in development of new AI systems.
news.harvard.edu
April 17, 2025 at 4:54 AM
For more, see Sonja's talk, or this recent article: news.harvard.edu/gazette/stor...
The artificial fish also behave in many ways, both obvious and subtle, like real weakly electric fish. This suggests that we might learn something about the neural computations of real electric fish by studying these artificial ones.
April 17, 2025 at 4:54 AM
The artificial fish also behave in many ways, both obvious and subtle, like real weakly electric fish. This suggests that we might learn something about the neural computations of real electric fish by studying these artificial ones.
By training these agents via multi-agent RL, she obtains interesting emergent behavior, including emergent patterns of social interaction. For example, fish can learn to 'freeload', or reduce their own electric output to exploit the output of other fish.
April 17, 2025 at 4:54 AM
By training these agents via multi-agent RL, she obtains interesting emergent behavior, including emergent patterns of social interaction. For example, fish can learn to 'freeload', or reduce their own electric output to exploit the output of other fish.
The model includes a fish brain (a recurrent neural network) that can learn from experience, and a detailed model of sensing via electric fields. Sonja et al. train these artificial fish to forage for food, and importantly, to do so in the presence of many other hungry fish!
April 17, 2025 at 4:54 AM
The model includes a fish brain (a recurrent neural network) that can learn from experience, and a detailed model of sensing via electric fields. Sonja et al. train these artificial fish to forage for food, and importantly, to do so in the presence of many other hungry fish!
Although we want understand the complex behavior of these fish in naturalistic environments, actually doing this remains challenging. Sonja et al. have partly bypassed this difficulty by developing a realistic computational model of these fish.
April 17, 2025 at 4:54 AM
Although we want understand the complex behavior of these fish in naturalistic environments, actually doing this remains challenging. Sonja et al. have partly bypassed this difficulty by developing a realistic computational model of these fish.
Weakly electric fish like Gnathonemus petersii have the ability to use electric fields in a variety of ways, including to sense their surroundings, as a means of attack, and as a means to communicate with other electric fish.
April 17, 2025 at 4:54 AM
Weakly electric fish like Gnathonemus petersii have the ability to use electric fields in a variety of ways, including to sense their surroundings, as a means of attack, and as a means to communicate with other electric fish.
Her work suggests that known model-based algorithms aren't the only route to rapid adaptation, and that this capability of animals might usefully be viewed from a meta-RL perspective.
April 17, 2025 at 4:29 AM
Her work suggests that known model-based algorithms aren't the only route to rapid adaptation, and that this capability of animals might usefully be viewed from a meta-RL perspective.
Using a transformer as a model system, Ching's work shows that key-value memory systems can in-context learn to navigate novel environments, accurately and in relatively few trials, given examples of previous navigation decisions.
April 17, 2025 at 4:29 AM
Using a transformer as a model system, Ching's work shows that key-value memory systems can in-context learn to navigate novel environments, accurately and in relatively few trials, given examples of previous navigation decisions.
Ching's work takes inspiration from the hippocampus, which some theoretical work portrays as a "key-value memory system". The idea is that animals may have learned to navigate *other* environments, and could in principle leverage these memories to navigate new spaces.
April 17, 2025 at 4:29 AM
Ching's work takes inspiration from the hippocampus, which some theoretical work portrays as a "key-value memory system". The idea is that animals may have learned to navigate *other* environments, and could in principle leverage these memories to navigate new spaces.
Algorithms which are either explicitly model-based, or that are at least 'more' model-based than TD learning (e.g., the successor representation), help agents learn to navigate a new environment much more quickly. But the neural correlates of these algorithms remain somewhat unclear.
April 17, 2025 at 4:29 AM
Algorithms which are either explicitly model-based, or that are at least 'more' model-based than TD learning (e.g., the successor representation), help agents learn to navigate a new environment much more quickly. But the neural correlates of these algorithms remain somewhat unclear.
When presented with a new environment (e.g., a maze), real animals can learn to navigate fairly quickly. This is not true of the simplest RL algorithm, TD learning, which would require a large number of experiences in that environment and may not generalize between environments.
April 17, 2025 at 4:29 AM
When presented with a new environment (e.g., a maze), real animals can learn to navigate fairly quickly. This is not true of the simplest RL algorithm, TD learning, which would require a large number of experiences in that environment and may not generalize between environments.
For more, see Kevin et al.'s paper, now published in Nature Neuroscience: www.nature.com/articles/s41...
The role of motor cortex in motor sequence execution depends on demands for flexibility - Nature Neuroscience
Motor cortex is required for flexible motor sequences informed by sensory cues but not for single automatic sequences. However, training automatic sequences alongside flexible ones interferes with the...
www.nature.com
March 5, 2025 at 11:34 PM
For more, see Kevin et al.'s paper, now published in Nature Neuroscience: www.nature.com/articles/s41...
But they also find that flexible sequence performance requirements can interfere with overtrained sequence performance requirements, at least when there is some similarity in context.
March 5, 2025 at 11:34 PM
But they also find that flexible sequence performance requirements can interfere with overtrained sequence performance requirements, at least when there is some similarity in context.
But is there a difference between motor sequences which require on-the-fly decisions about motor actions (e.g., based on instructive cues), and heavily overtrained ones? Kevin et al. find that the answer is yes! Motor cortex appears to be required for the former, but not the latter.
March 5, 2025 at 11:34 PM
But is there a difference between motor sequences which require on-the-fly decisions about motor actions (e.g., based on instructive cues), and heavily overtrained ones? Kevin et al. find that the answer is yes! Motor cortex appears to be required for the former, but not the latter.
Motor cortex is thought to be required for motor learning, but not necessarily performance; it is known that if mice practice the same motor sequences over and over, lesioning motor cortex mostly doesn't affect the performance of those heavily overtrained sequences.
March 5, 2025 at 11:34 PM
Motor cortex is thought to be required for motor learning, but not necessarily performance; it is known that if mice practice the same motor sequences over and over, lesioning motor cortex mostly doesn't affect the performance of those heavily overtrained sequences.
Relatedly, if you only ever practice a piece of music one way, it can become hard to modify it midway through. In brains, one mechanism thought to be responsible is the interplay between the motor cortex and subcortical motor circuits.
March 5, 2025 at 11:34 PM
Relatedly, if you only ever practice a piece of music one way, it can become hard to modify it midway through. In brains, one mechanism thought to be responsible is the interplay between the motor cortex and subcortical motor circuits.
As we learn by trial and error to perform rewarded actions, it is known that the mechanisms underlying that action performance change as well. Think about how, if you practice a piece of music for a long time, it can become somewhat automatic.
March 5, 2025 at 11:34 PM
As we learn by trial and error to perform rewarded actions, it is known that the mechanisms underlying that action performance change as well. Think about how, if you practice a piece of music for a long time, it can become somewhat automatic.