Anton Ehrmanntraut
@aehrm.bsky.social
doing NLP, statistics, coding, and similar stuff;
Computational Literary Studies @ Universität Würzburg;
https://anton.ehrmanntraut.de
Computational Literary Studies @ Universität Würzburg;
https://anton.ehrmanntraut.de
Reposted by Anton Ehrmanntraut
One of the questions we debated while training ModernBERT was whether a modern trained encoder would unlock zero-shot reasoning using only it's generative head?
Spoilers: the answer is yes.
Spoilers: the answer is yes.
![from transformers import pipeline
model_name = "answerdotai/ModernBERT-Large-Instruct"
fill_mask = pipeline("fill-mask", model=model_name, tokenizer=model_name)
text = """You will be given a question and options. Select the right answer.
QUESTION: If (G, .) is a group such that (ab)^-1 = a^-1b^-1, for all a, b in G, then G is a/an
CHOICES:
- A: commutative semi group
- B: abelian group
- C: non-abelian group
- D: None of these
ANSWER: [unused0] [MASK]"""
results = fill_mask(text)
answer = results[0]["token_str"].strip()
print(f"Predicted answer: {answer}") # Answer: B](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:psfkl7gi24rg5rhvv7z6mly3/bafkreid6ko23ckgz2mwpp7zj3p6n333vmg5c2r5bghon77g4el6322z6oa@jpeg)
February 10, 2025 at 6:13 PM
One of the questions we debated while training ModernBERT was whether a modern trained encoder would unlock zero-shot reasoning using only it's generative head?
Spoilers: the answer is yes.
Spoilers: the answer is yes.