So far we have seen almost 37,000 User-Agents mentioned in files on more than 600 million hostnames!

openrobotstxt.org

Hi @johnmu.com we'd like to extend our HREFlang integration to include data attributes in the source code. Can Google handle the possibility of additional information in the HTML element with hreflang links that Googlebot can simply ignore?

Hi @johnmu.com , I've blocked a domain directory in my robots.txt, which Google is crawling unnecessarily because it's my site search and everything is set to noindex.

As @tentaclequing.bsky.social and @rustybrick.com has already reported, there are increasing problems with the Googlebot, which thinks a page is very slow and then dramatically reduces crawling.

Looking for some SEO peeps to follow on Bluesky?

Here's a Starter Pack for you: go.bsky.app/NK1jwq3

Great article from @cyrusshepard.bsky.social about the true Google index size:

zyppy.com/seo/google-i...