Thrilled to see this paper finally published! The manifold representations in Figure 1H were a key inspiration behind the visualization of Successor Features (SF) in our recent SF paper. It's fascinating how insights from neuroscience can help us interpret representations learned in deep RL.

If you are at NeurIPS, I will be presenting the poster this Thursday, 12 Dec, 4:30 p.m. PST — 7:30 p.m at West Ballroom A-D #6504. See you there :)

The next study is an example of the benefits of the peer review process! Thanks to an anonymous reviewer, we trained a non-linear decoder to evaluate how well the learned SFs can be transformed into SRs. Our approach showed consistently lower mean squared errors.
10/11

How well do learned SFs capture environment statistics? Using geospatial color mapping, we found that Simple SFs form organized clusters in neural space, mirroring physical proximity and capturing key environmental dynamics.
9/11

Does it scale to complex tasks? Absolutely! We demonstrated the effectiveness of Simple SFs in Mujoco, using continuous control tasks to test its robustness and adaptability in complex environments. Importantly, all our studies were conducted using pixel observations only.
8/11

To demonstrate SFs' generalization, we tested Simple SFs in a continual RL setting: 2D Minigrid with changing dynamics/reward locations, and 3D Four Rooms with alternating rewards (-1/+1) per task switch. Here, we show only the 3D Four Rooms results.
7/11

How do we learn? Simple SFs use two key losses: Reward Prediction for the task encoding vector and Q-SF-TD for learning SFs. Basis features (Φ) are normalized outputs from the Encoder, with a stop-gradient to prevent interference in task encoding.
6/11

To address this, we developed Simple Successor Features. By preserving the linear relationship among SFs (Ψ), the task encoding vector (w), and Q-values, our approach prevents representation collapse and enables efficient learning from pixels.
5/11

Background: The canonical SF-TD loss is a common approach for learning SFs. But optimizing both basis features (Φ) and SFs (Ψ) concurrently can cause representation collapse, as setting them to constants becomes a trivial solution.
4/11

But my favorite moment got to be enjoying Starbucks coffee with my lab @tyrellturing.bsky.social , @oliviercodol.bsky.social , Roman Pogodin as we celebrated our arrival in the Big 🍎. A perfect blend of science, caffeine, and NYC energy!

Overall, NAISys felt like an extended summer school - small enough for meaningful interactions and focused on diverse NeuroAI topics. Kudos to the organizers, @tonyzador.bsky.social , Doris Tsao and @tyrellturing.bsky.social for an amazing event!

A highlight of the conference was meeting people passionate about interdisciplinary research, especially during the poster session. It was great to connect with @computingnature.neuromatch.social.ap.brid.gy in person after our online @neuromatch.bsky.social interactions.

This past week, I had a fantastic time attending NAISys at
@cshlaboratory.bsky.social - a NeuroAI conference. It was amazing to learn about various topics like genomic coding, neuromorphic hardware, and in-context learning at the home to eight Nobel Prize winners in Physiology or Medicine! 1/N

Bluesky now has over 10 million users, and I was #4,792,410!