Matteo Tiezzi
@mtiezzi.bsky.social
PostDoc Researcher @ IIT, Continual and Lifelong Learning -> Robots, Graph Neural Networks, Sequence Processing | CoLLAs 2024 Local Chair
🏠 mtiezzi.github.io
🏠 mtiezzi.github.io
(Top) A standard MLP catastrophically forgets samples learned online when data distribution shifts
(Bottom) A Memory Head is capable to avoid forgetting thanks to its learnable key-value routing mechanism. Keys become representative of the data distribution.
proceedings.mlr.press/v274/tiezzi2...
(Bottom) A Memory Head is capable to avoid forgetting thanks to its learnable key-value routing mechanism. Keys become representative of the data distribution.
proceedings.mlr.press/v274/tiezzi2...
February 19, 2025 at 2:27 PM
(Top) A standard MLP catastrophically forgets samples learned online when data distribution shifts
(Bottom) A Memory Head is capable to avoid forgetting thanks to its learnable key-value routing mechanism. Keys become representative of the data distribution.
proceedings.mlr.press/v274/tiezzi2...
(Bottom) A Memory Head is capable to avoid forgetting thanks to its learnable key-value routing mechanism. Keys become representative of the data distribution.
proceedings.mlr.press/v274/tiezzi2...
TL;DR: in Memory Heads, neurons route their computation across a learnable key-value mechanism
⚡️ dynamic behaviour depending on their own input.
⚡ Only the units(weights) that are relevant to process the observed sample are blended ➡ parameter isolation
proceedings.mlr.press/v274/tiezzi2...
⚡️ dynamic behaviour depending on their own input.
⚡ Only the units(weights) that are relevant to process the observed sample are blended ➡ parameter isolation
proceedings.mlr.press/v274/tiezzi2...
February 19, 2025 at 2:27 PM
TL;DR: in Memory Heads, neurons route their computation across a learnable key-value mechanism
⚡️ dynamic behaviour depending on their own input.
⚡ Only the units(weights) that are relevant to process the observed sample are blended ➡ parameter isolation
proceedings.mlr.press/v274/tiezzi2...
⚡️ dynamic behaviour depending on their own input.
⚡ Only the units(weights) that are relevant to process the observed sample are blended ➡ parameter isolation
proceedings.mlr.press/v274/tiezzi2...
@logconference.bsky.social #LOG meetup with @federico-errica.bsky.social open talk on "What is going on with oversmoothing, oversquashing, and underreaching?"
December 6, 2024 at 1:07 PM
@logconference.bsky.social #LOG meetup with @federico-errica.bsky.social open talk on "What is going on with oversmoothing, oversquashing, and underreaching?"
