The key idea is to split up context encoding from query processing/token generation. The context encoding stage is then split across multiple blocks for parallel computation.

There's a tunable parameter that lets you balance the tradeoff between accuracy and speed. And best of all, it operates on a different mechanism from other popular LLM optimization techniques like Flash Attention and KV cache compression meaning that you can combine them for more speed improvements.

but are they too hot?

for now

true but is rothfuss ever going to finish it? i’d love you to be right!

this seems sadly most likely to me too