Manvi Agarwal
@manviagarwal.bsky.social
3️⃣ We explain why structural information in PE has proven to be empirically successful.
We go back to 1️⃣ and use the content-context connection to show that the higher is the mutual information between data and its positional representation, the better is task performance.
We go back to 1️⃣ and use the content-context connection to show that the higher is the mutual information between data and its positional representation, the better is task performance.
April 8, 2025 at 4:52 AM
3️⃣ We explain why structural information in PE has proven to be empirically successful.
We go back to 1️⃣ and use the content-context connection to show that the higher is the mutual information between data and its positional representation, the better is task performance.
We go back to 1️⃣ and use the content-context connection to show that the higher is the mutual information between data and its positional representation, the better is task performance.
2️⃣ We introduce a new positional encoding method - RoPEPool - that can model causality.
How does RoPEPool compare to RoPE and F-StrIPE? Our analysis with a toy example says: RoPEPool isn’t just different, it’s also richer in terms of expressivity.
How does RoPEPool compare to RoPE and F-StrIPE? Our analysis with a toy example says: RoPEPool isn’t just different, it’s also richer in terms of expressivity.
April 8, 2025 at 4:52 AM
2️⃣ We introduce a new positional encoding method - RoPEPool - that can model causality.
How does RoPEPool compare to RoPE and F-StrIPE? Our analysis with a toy example says: RoPEPool isn’t just different, it’s also richer in terms of expressivity.
How does RoPEPool compare to RoPE and F-StrIPE? Our analysis with a toy example says: RoPEPool isn’t just different, it’s also richer in terms of expressivity.
1️⃣ We show how different families of positional encoding - rotation-based (RoPE) and random fourier features-based (F-StrIPE) - can be compared using kernel methods.
It’s not just vibes - we characterize precisely how queries and keys are affected by positional information.
It’s not just vibes - we characterize precisely how queries and keys are affected by positional information.
April 8, 2025 at 4:52 AM
1️⃣ We show how different families of positional encoding - rotation-based (RoPE) and random fourier features-based (F-StrIPE) - can be compared using kernel methods.
It’s not just vibes - we characterize precisely how queries and keys are affected by positional information.
It’s not just vibes - we characterize precisely how queries and keys are affected by positional information.
ICASSP 2025 has begun! I will be presenting "F-StrIPE: Fast Structure-Informed Positional Encoding for Symbolic Music Generation" on April 11 in the lecture session "Machine learning for speech, audio and music processing II" at 2 PM.
More details below 🧵
More details below 🧵
April 7, 2025 at 11:48 AM
ICASSP 2025 has begun! I will be presenting "F-StrIPE: Fast Structure-Informed Positional Encoding for Symbolic Music Generation" on April 11 in the lecture session "Machine learning for speech, audio and music processing II" at 2 PM.
More details below 🧵
More details below 🧵
