lastphlogistonist
LightNews LightNews
rhophi.bsky.social
lastphlogistonist
@rhophi.bsky.social
5 followers 39 following 2 posts
Interested in math, philosophy, physics, computation, complex systems, consciousness, and altruism.
Posts Replies Media Videos
Reposted by lastphlogistonist
tedunderwood.com
Did you know that attention across the whole input span was inspired by the time-negating alien language in Arrival? Crazy anecdote from the latest Hard Fork podcast (by @kevinroose.com and @caseynewton.bsky.social). HT nwbrownboi on Threads for the lead.
Transcript of Hard Fork ep 111: Yeah. And I could talk for an hour about transformers and why they are so important. But I think it's important to say that they were inspired by the alien language in the film Arrival, which had just recently come out. And a group of researchers at Google, one researcher in particular, who was part of that original team, was inspired by watching Arrival and seeing that the aliens in the movie had this language which represented entire sentences with a single symbol. And they thought, hey, what if we did that inside of a neural network? So rather than processing all of the inputs that you would give to one of these systems one word at a time, you could have this thing called an attention mechanism, which paid attention to all of it simultaneously. That would allow you to process much more information much faster. And that insight sparked the creation of the transformer, which led to all the stuff we see in Al today.
December 1, 2024 at 2:50 PM
Reposted by lastphlogistonist
seanmcarroll.bsky.social
We laugh, but: a clock is something that repeats reliably with respect to other clocks.
Clocks
November 26, 2024 at 5:40 PM