4/ The one difference - all of the checks now have an extra zero (or two)

“Even young PhDs could pull a half million dollars a year”

Ilya was offered “nearly $2 million for the first year” in 2016

Some things do change!

3/ Another thing that remains unchanged after a decade and a half - Mark Zuckerberg is extremely uninterested in AI safety, ie why DeepMind refused to sell out to Facebook

2/ It is uncanny how much Meta’s efforts in the last year have just been a rerun of Mark’s original effort to build FAIR by hiring Yann Lecun back in 2013

- Invitations to 1-on-1 dinners with Mark

- A desk next to Mark at the office

- “Intelligent agents” booking airline tickets

- “Open source”

12/ In that vein, Genius Makers is the apotheosis of the "stamp collecting" era of ML research

Let's hope we're finally entering the physics era

11/ A friend recently told me a hilarious quote from Ernest Rutherford: "All science is either physics or stamp collecting"

If you can't derive some underlying principle of reality from your research, then ultimately you're just collecting a bunch of random facts to no end

10/ Page 72 literally mentions Andrew Ng showing a literal line graph to Larry Page in 2010 charting how multimodal ML performance would consistently improve with the application of more training data

But the book never *quite* brings it all together

9/ Page 70 mentions an eyebrow-raising anecdote about how one of Geoff Hinton's grad students discovered at Google in 2011 that training a speech model with 2000 hours of data instead of 12 hours would miraculously improve error rates

8/ The result is that the most uncanny part of reading Genius Makers is when you see the ghost of scaling laws looming on the edges

On Page 50, the book mentions Terry Sejnowski's 1987 NETtalk paper, which arguably plotted out the world's first log-linear AI scaling law

7/ But back in the late 2010s, I think it's pretty clear that Cade Metz was still documenting a pre-paradigmatic science, with no consensus on the 'grand unifying theory' that would drive ML research forward

6/ Today, I think it's probably fair to say frontier AI research is on the cusp of becoming a "normal science" for the first time, as the field has coalesced around the scaling laws paradigm that ML performance can improve predictably with the application of more compute

5/ "Normal" science = developed fields where researchers try to solve puzzles within an established paradigm

"Pre-paradigmatic" science = emerging fields with competing schools of thought, no consensus, and disagreement on what problems are even worth studying in the first place

4/ But as much as I like to make fun of NYT journalists, I think there's a more fundamental explanation for where the book went wrong - that goes back to the philosopher Thomas Kuhn's old distinction between "pre-paradigmatic" science and "normal" science

3/ The result? The book spends hundreds of pages talking about: AlphaGo, GANs, LSTMs, RL agents playing DOTA...

Zero mentions of: GPT-2

In fact, there's only a *single* mention of the word "transformer" in the entire 300 page body of the book (a one-off reference to BERT)

2/ The simple explanation is that Genius Makers is a history book that ends right before all the crazy stuff happens

The book was published in March 2021, meaning the final draft was probably finished in the summer of 2020. GPT-3 came out in May 2020

Basically success for a startup is finding the intersection in the Venn diagram between “sounds crazy to any rational person” and “actually works.” The problem is most of the time you’re just, yknow, actually crazy

In 1978, AT&T launched the US's first modern cell service in Chicago. The nationwide launch was scheduled for the early 80s, but never happened because AT&T was broken up for antitrust violations in 1982

Predicting the future is easy. Making money is hard

Like, every single part of this sentence is wrong?? Inference on a 500M parameter model requires 1 billion flops, which is not 1000 flops, which is also not 1 tflop (that's a trillion flops)

LLMs are actually fairly good at explaining how they work these days... try asking them!