Anthony Fuller
@anthonyfuller.bsky.social
PhD Student at Carleton University (Ottawa, Canada)
https://antofuller.github.io/
https://antofuller.github.io/
Lookhere position encoding could help with this: arxiv.org/abs/2405.13985
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate
High-resolution images offer more information about scenes that can improve model accuracy. However, the dominant model architecture in computer vision, the vision transformer (ViT), cannot effectivel...
arxiv.org
February 27, 2025 at 5:04 PM
Lookhere position encoding could help with this: arxiv.org/abs/2405.13985
Awesome, thanks for the explanation!
February 4, 2025 at 4:55 PM
Awesome, thanks for the explanation!
Cool work? I think the position encoding method looks similar to DeBERTa’s disentangled attention: arxiv.org/abs/2006.03654
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBE...
arxiv.org
February 4, 2025 at 2:51 PM
Cool work? I think the position encoding method looks similar to DeBERTa’s disentangled attention: arxiv.org/abs/2006.03654