Jannis Vamvas
@vamvas.bsky.social
Natural Language Processing (#NLProc #NLP) Researcher @ University of Zurich @cl-uzh.bsky.social
vamvas.ch
vamvas.ch
I also found what others complained about previously: The “Predicted Outputs” feature is unstable even with temp=0.
It’s unclear to me why this happens – do they batch the verification steps of users? An artifact of temperature sampling?
So, any hints would be appreciated! 4/4
It’s unclear to me why this happens – do they batch the verification steps of users? An artifact of temperature sampling?
So, any hints would be appreciated! 4/4
April 21, 2025 at 4:52 PM
I also found what others complained about previously: The “Predicted Outputs” feature is unstable even with temp=0.
It’s unclear to me why this happens – do they batch the verification steps of users? An artifact of temperature sampling?
So, any hints would be appreciated! 4/4
It’s unclear to me why this happens – do they batch the verification steps of users? An artifact of temperature sampling?
So, any hints would be appreciated! 4/4
Systematic probing of the OpenAI API reveals that they are using a lookahead parameter of K=16, and perform prefix matching with a prefix threshold of 32 tokens.
A simulation experiment confirms that the reverse-engineered algorithm matches the behavior of the API. 3/4
A simulation experiment confirms that the reverse-engineered algorithm matches the behavior of the API. 3/4
April 21, 2025 at 4:52 PM
Systematic probing of the OpenAI API reveals that they are using a lookahead parameter of K=16, and perform prefix matching with a prefix threshold of 32 tokens.
A simulation experiment confirms that the reverse-engineered algorithm matches the behavior of the API. 3/4
A simulation experiment confirms that the reverse-engineered algorithm matches the behavior of the API. 3/4
Draft verification is a well-known idea in NLP, from Speculative Decoding to algorithms leveraging a static draft, such as Aggressive Decoding and Prompt Lookup Decoding.
Clearly, “Predicted Outputs” does something similar. But how does it work, specifically? 2/4
Clearly, “Predicted Outputs” does something similar. But how does it work, specifically? 2/4
April 21, 2025 at 4:52 PM
Draft verification is a well-known idea in NLP, from Speculative Decoding to algorithms leveraging a static draft, such as Aggressive Decoding and Prompt Lookup Decoding.
Clearly, “Predicted Outputs” does something similar. But how does it work, specifically? 2/4
Clearly, “Predicted Outputs” does something similar. But how does it work, specifically? 2/4
OpenAI introduced the “Predicted Outputs” feature last November, but their documentation raises more questions than it answers. So let’s reverse-engineer the feature! 🧵1/4
platform.openai.com/docs/guides/...
platform.openai.com/docs/guides/...
OpenAI Platform
Explore developer resources, tutorials, API docs, and dynamic examples to get the most out of OpenAI's platform.
platform.openai.com
April 21, 2025 at 4:52 PM
OpenAI introduced the “Predicted Outputs” feature last November, but their documentation raises more questions than it answers. So let’s reverse-engineer the feature! 🧵1/4
platform.openai.com/docs/guides/...
platform.openai.com/docs/guides/...