Massive shoutout to the efforts of Sanghyeok Choi (he's on the dark MAGA app), Salem Lahlou (mbzuai.ac.ae/study/facult...), and ‪‪@oyounis.bsky.social - this was very much a team effort and we’re really excited to help popularize gflownet use through these tools. We really value your feedback!

If you’re interested in using torchgfn, helping us improve the library, want help incorporating torchgfn into your workflow, or have any feedback, please feel free to familiarize yourself with our documentation and reach out – there’s still lots to do!

torchgfn.readthedocs.io/en/latest/

Moving forward, we plan to focus on optimizing the library for large-scale distributed training setups, and supporting more specialized and demanding environments, particularly in the AI for Science domains.

The structure of the GFlowNet itself is highly modular, permitting the use of modified losses, custom samplers, novel off-policy sampling methods, and new policy architectures with minimal changes to the underlying library elements.

Basic torchgfn usage follows standard pytorch workflows, allowing the user to swap in any modified components to support the development of new methods:

This is a major update:

+ Much easier environment definition.
+ Cleaner abstractions → easier extensibility!
+ Support for graph-based states under torch_geometric.
+ Improvements to every core element of the library.
+ Lots of new environments, tutorials, and examples!

Yeah the lack of memepoasters and tpot adjacent attention bait makes the platform great for us but bad for it taking over the disinformation psyop town square.

@boussifo.bsky.social , Léna Néhale Ezzine, Michał Koziarski, @jainmoksh.bsky.social , Nikolay Malkin, @bengioe.bsky.social , Rim Assouel, @yoshuabengio.bsky.social - @mila-quebec.bsky.social

Action Abstractions for Amortized Sampling
📄 arxiv.org/abs/2410.15184
💻 github.com/GFNOrg/Chunk...

The additive benefit of combining chunking with diversity-seeking samplers, like GFlowNets, also points towards an intriguing explanation as to why macro action discovery has not been found generally useful in the RL context.

💡 Why is this exciting?

Hierarchical planning is a key component of intelligence—both biological & artificial. By dynamically learning & using abstractions, our method bridges the gap between RL, program induction, and cognitive science.

These chunks also generalize - they’re transferable across samplers and tasks!

Chunks learned in one environment improve exploration and sampling efficiency in unseen settings, suggesting the method abstracts high order general principles that are robust & adaptable to new envs!

For mode discovery, our approach also significantly speeds up discovering diverse high-reward states.

For example, in FractalGrid, vanilla GFlowNets get stuck in a single mode, but armed with ActionPiece, it unlocks new exploration paths!

Chunking helps!

Across synthetic and real-world tasks (e.g., RNA sequence generation, bit sequences, and FractalGrid), our approach improves especially for GFlowNets:

✅ Mode discovery
✅ Exploration
✅ Density estimation
✅ Interpretability

By applying BPE (which we're calling "ActionPiece" for learning chunked actions) to sampled trajectories, we extract meaningful high-level actions that naturally emerge during learning. For example, here are some learned chunks from sampler of RNA binders:

We chunk frequently occurring subsequences into high-order actions using Byte Pair Encoding (BPE)—a popular NLP tokenization technique. These chunks are added to the action space, which progressively reduces trajectory length and helps uncover latent structures in the task.

In RL & GFlowNets, with longer trajectories, assigning credit and discovering diverse high-reward states gets harder. Standard methods struggle to sample structured distributions efficiently & many previous attempts to discover high-order actions failed to show consistent benefit.