I found the above link on Reddit. Here’s another interesting related video. www.youtube.com/watch?v=0LZ2...

With current synthetic data techniques, one issue is that they don’t add a lot of new entropy to the original pre-training data. A possible solution is to spend more compute time during testing to generate synthetic data with higher entropy content!