Birchlabs
@birchlabs.co.uk
ML Engineer at Anlatan (NovelAI). co-author of HDiT (Hourglass Diffusion Transformers). works on diffusion models and LLMs. 日本語を勉強してる。
pytorch 2.7 is out!
- Mega Cache looks nice, unclear whether those modules get cached by legacy mechanisms too
- foreach map looks good for optimizers
- trainable biases means you can now train T5 on flex
- prologue fusion is hype
- context parallel brings ring attention
pytorch.org/blog/pytorch...
- Mega Cache looks nice, unclear whether those modules get cached by legacy mechanisms too
- foreach map looks good for optimizers
- trainable biases means you can now train T5 on flex
- prologue fusion is hype
- context parallel brings ring attention
pytorch.org/blog/pytorch...
April 24, 2025 at 2:38 PM
pytorch 2.7 is out!
- Mega Cache looks nice, unclear whether those modules get cached by legacy mechanisms too
- foreach map looks good for optimizers
- trainable biases means you can now train T5 on flex
- prologue fusion is hype
- context parallel brings ring attention
pytorch.org/blog/pytorch...
- Mega Cache looks nice, unclear whether those modules get cached by legacy mechanisms too
- foreach map looks good for optimizers
- trainable biases means you can now train T5 on flex
- prologue fusion is hype
- context parallel brings ring attention
pytorch.org/blog/pytorch...
pytorch 2.6 is out!
highlights:
- flex attention: better compilation of blockmask creation, better support for dynamic shapes
- cuDNN SDPA: fixes for memory layout
- CUDA 12.6
- python 3.13
- MaskedTensor memory leak fix
highlights:
- flex attention: better compilation of blockmask creation, better support for dynamic shapes
- cuDNN SDPA: fixes for memory layout
- CUDA 12.6
- python 3.13
- MaskedTensor memory leak fix
January 30, 2025 at 12:41 AM
pytorch 2.6 is out!
highlights:
- flex attention: better compilation of blockmask creation, better support for dynamic shapes
- cuDNN SDPA: fixes for memory layout
- CUDA 12.6
- python 3.13
- MaskedTensor memory leak fix
highlights:
- flex attention: better compilation of blockmask creation, better support for dynamic shapes
- cuDNN SDPA: fixes for memory layout
- CUDA 12.6
- python 3.13
- MaskedTensor memory leak fix
pytorch 2.6 final RC is out, promoting to stable in a couple of days!
mostly I'm looking forward to better compilation of flex block mask creation, and better support for flex attention on dynamic shapes.
there's also fixes for memory layout in cuDNN SDPA.
dev-discuss.pytorch.org/t/pytorch-re...
mostly I'm looking forward to better compilation of flex block mask creation, and better support for flex attention on dynamic shapes.
there's also fixes for memory layout in cuDNN SDPA.
dev-discuss.pytorch.org/t/pytorch-re...
January 20, 2025 at 7:27 PM
pytorch 2.6 final RC is out, promoting to stable in a couple of days!
mostly I'm looking forward to better compilation of flex block mask creation, and better support for flex attention on dynamic shapes.
there's also fixes for memory layout in cuDNN SDPA.
dev-discuss.pytorch.org/t/pytorch-re...
mostly I'm looking forward to better compilation of flex block mask creation, and better support for flex attention on dynamic shapes.
there's also fixes for memory layout in cuDNN SDPA.
dev-discuss.pytorch.org/t/pytorch-re...
drink cups should put the hole in the bottom.
heat rises. "the top is cool enough to drink" implies "everything below it is colder".
drinking from the bottom lets us access safe temperatures earlier and before the whole cup cools.
heat rises. "the top is cool enough to drink" implies "everything below it is colder".
drinking from the bottom lets us access safe temperatures earlier and before the whole cup cools.
January 4, 2025 at 11:37 PM
drink cups should put the hole in the bottom.
heat rises. "the top is cool enough to drink" implies "everything below it is colder".
drinking from the bottom lets us access safe temperatures earlier and before the whole cup cools.
heat rises. "the top is cool enough to drink" implies "everything below it is colder".
drinking from the bottom lets us access safe temperatures earlier and before the whole cup cools.
running npm version from a subdirectory of a git repository is literally an unsolved problem in 2024
github.com/npm/cli/issu...
github.com/npm/cli/issu...
January 4, 2025 at 8:37 PM
running npm version from a subdirectory of a git repository is literally an unsolved problem in 2024
github.com/npm/cli/issu...
github.com/npm/cli/issu...
I should just get this tattooed, I never remember how to find it
pip install huggingface_hub[hf_transfer]
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download
pip install huggingface_hub[hf_transfer]
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download
January 1, 2025 at 11:50 PM
I should just get this tattooed, I never remember how to find it
pip install huggingface_hub[hf_transfer]
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download
pip install huggingface_hub[hf_transfer]
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download
when the standard library comments out std::experimental::observer_ptr just to stop you having fun
December 22, 2024 at 2:09 AM
when the standard library comments out std::experimental::observer_ptr just to stop you having fun
NovelAI v4 makes dreams come true
December 21, 2024 at 9:49 PM
NovelAI v4 makes dreams come true
when you're measuring torch compile warmup "oh 11 secs that's not so bad"
then you realize it was 111 secs
then you realize it was 111 secs
December 19, 2024 at 6:00 PM
when you're measuring torch compile warmup "oh 11 secs that's not so bad"
then you realize it was 111 secs
then you realize it was 111 secs
if you care about multiprocess debugging in VSCode please upvote this issue
so we don't have to click terminate a hundred times
github.com/microsoft/vs...
so we don't have to click terminate a hundred times
github.com/microsoft/vs...
December 16, 2024 at 10:26 PM
if you care about multiprocess debugging in VSCode please upvote this issue
so we don't have to click terminate a hundred times
github.com/microsoft/vs...
so we don't have to click terminate a hundred times
github.com/microsoft/vs...
Meta releases flow-matching code
github.com/facebookrese...
github.com/facebookrese...
GitHub - facebookresearch/flow_matching: A PyTorch library for implementing flow matching algorithms, featuring continuous and discrete flow matching implementations. It includes practical examples fo...
A PyTorch library for implementing flow matching algorithms, featuring continuous and discrete flow matching implementations. It includes practical examples for both text and image modalities. - fa...
github.com
December 14, 2024 at 3:48 AM
Meta releases flow-matching code
github.com/facebookrese...
github.com/facebookrese...
the EDM2 repository had an Autoguidance update 4 days ago demonstrating their NeurIPS Oral paper "Guiding a Diffusion Model with a Bad Version of Itself"
github.com/NVlabs/edm2
github.com/NVlabs/edm2
GitHub - NVlabs/edm2: EDM2 and Autoguidance -- Official PyTorch implementation
EDM2 and Autoguidance -- Official PyTorch implementation - NVlabs/edm2
github.com
December 13, 2024 at 12:25 AM
the EDM2 repository had an Autoguidance update 4 days ago demonstrating their NeurIPS Oral paper "Guiding a Diffusion Model with a Bad Version of Itself"
github.com/NVlabs/edm2
github.com/NVlabs/edm2
torch profiler record_function spans are not free, even when you're not profiling a model. the model I'm benchmarking trained 2.5% faster when I commented them all out.
December 12, 2024 at 6:23 PM
torch profiler record_function spans are not free, even when you're not profiling a model. the model I'm benchmarking trained 2.5% faster when I commented them all out.
Box2D 3.0.0 in WebAssembly+TypeScript starting to work
December 9, 2024 at 1:32 AM
Box2D 3.0.0 in WebAssembly+TypeScript starting to work
Reposted by Birchlabs
🤔 Why do we extract diffusion features from noisy images? Isn’t that destroying information?
Yes, it is - but we found a way to do better. 🚀
Here’s how we unlock better features, no noise, no hassle.
📝 Project Page: compvis.github.io/cleandift
💻 Code: github.com/CompVis/clea...
🧵👇
Yes, it is - but we found a way to do better. 🚀
Here’s how we unlock better features, no noise, no hassle.
📝 Project Page: compvis.github.io/cleandift
💻 Code: github.com/CompVis/clea...
🧵👇
December 4, 2024 at 11:31 PM
🤔 Why do we extract diffusion features from noisy images? Isn’t that destroying information?
Yes, it is - but we found a way to do better. 🚀
Here’s how we unlock better features, no noise, no hassle.
📝 Project Page: compvis.github.io/cleandift
💻 Code: github.com/CompVis/clea...
🧵👇
Yes, it is - but we found a way to do better. 🚀
Here’s how we unlock better features, no noise, no hassle.
📝 Project Page: compvis.github.io/cleandift
💻 Code: github.com/CompVis/clea...
🧵👇
torch.compile is hard for dynamic shapes / large number of static shapes, and non-transformer architectures.
I measure suites of shapes, log recompiles, check which require warmup.
operation compile competitive with whole-model compile.
some operations prefer compiler disabled.
dynamic often slow.
I measure suites of shapes, log recompiles, check which require warmup.
operation compile competitive with whole-model compile.
some operations prefer compiler disabled.
dynamic often slow.
December 4, 2024 at 1:22 AM
torch.compile is hard for dynamic shapes / large number of static shapes, and non-transformer architectures.
I measure suites of shapes, log recompiles, check which require warmup.
operation compile competitive with whole-model compile.
some operations prefer compiler disabled.
dynamic often slow.
I measure suites of shapes, log recompiles, check which require warmup.
operation compile competitive with whole-model compile.
some operations prefer compiler disabled.
dynamic often slow.
Reposted by Birchlabs
A common question nowadays: Which is better, diffusion or flow matching? 🤔
Our answer: They’re two sides of the same coin. We wrote a blog post to show how diffusion models and Gaussian flow matching are equivalent. That’s great: It means you can use them interchangeably.
Our answer: They’re two sides of the same coin. We wrote a blog post to show how diffusion models and Gaussian flow matching are equivalent. That’s great: It means you can use them interchangeably.
December 2, 2024 at 6:45 PM
A common question nowadays: Which is better, diffusion or flow matching? 🤔
Our answer: They’re two sides of the same coin. We wrote a blog post to show how diffusion models and Gaussian flow matching are equivalent. That’s great: It means you can use them interchangeably.
Our answer: They’re two sides of the same coin. We wrote a blog post to show how diffusion models and Gaussian flow matching are equivalent. That’s great: It means you can use them interchangeably.
mood: adding unused variables to make torch inductor compile my triton kernel
November 30, 2024 at 9:53 PM
mood: adding unused variables to make torch inductor compile my triton kernel
that feel when you’ve been cooking rice for 10 minutes but the hob wasn’t turned on
November 28, 2024 at 10:45 PM
that feel when you’ve been cooking rice for 10 minutes but the hob wasn’t turned on
fine I'll use pytorch nightly
November 27, 2024 at 1:25 AM
fine I'll use pytorch nightly
born too late to learn maths from touhou pre-fight cutscenes
www.youtube.com/watch?v=tuDA...
www.youtube.com/watch?v=tuDA...
November 23, 2024 at 1:42 AM
born too late to learn maths from touhou pre-fight cutscenes
www.youtube.com/watch?v=tuDA...
www.youtube.com/watch?v=tuDA...