Particularly stiking example of biases in language models: The german short-bio generated by Safari says Virgina Woolf was a male author.

I would be very interested to see how their learning curve compares to „regular“ SGD

On some toy examples that I‘ve tested, I get quite a bit of points being predicted either in the ocean or in other unlikely places like Antarctica. To me this looks like the model hasnt yet learned a good posterior distribution on the globe. Do you monitor these unlikely predictions during training?