Setting aside the small person sample, randomizing the tasks within developer is....super odd? Wouldn't we start to expect a lot of order effects?
The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.
shotwell.ca/posts/2022-0...
shotwell.ca/posts/2022-0...