[6/6] The other idea is to do the weighted combination at an instance level. We look at intermediate layers for *each token* and slightly modify the overall distribution. This leads to consistent accuracy improvements for many models and datasets!

Would love to see some theory on why this works!

[5/6] Here's a nice example. We want to do some math. Greedy decoding leads to 5 x $10 = $50 for the overtime pay. This is cus A x B = C is a common pattern. But we really need A x B x C = D to get the answer. SLED can help with this because the internal layers happen to predict 'x' instead of '='.

[4/6] Our main decoding trick is to use a weighted combination of *all of the layers*. Precisely, we project the layers into the same output distribution (over vocab tokens). Then we combine the intermediate "logits" with the output logits based on our estimate of the LLM's internal knowledge

[3/6] The key observation is that LLMs "know" a lot more than they "tell" -- basically the training process can favor more popular tokens (in the dataset) rather than more accuracy predictions for the query at hand.

So we can utilize this during decoding time...

Longer thread about our new factuality decoding method SLED at NeurIPS 2024. Main idea: freeze the model, but be thoughtful about the decoding. With a small amount of extra inference-time compute, we increase accuracy by 3% on several benchmarks! SLED helps for all major open source models!

First shameless plug -- our new factuality decoding method, SLED gets SOTA improvements on 14+ models (Llama 2/3, Gemma, Mistral) & 9 benchmarks!

See our #NeurIPS2024 poster today (Friday) in the East Exhibit Hall A-C #3311