OpenRobotsTXT
LightNews LightNews
openrobotstxt.org
OpenRobotsTXT
@openrobotstxt.org
24 followers 1 following 3 posts
OpenRobotsTXT is an open archive of the world’s robots.txt files.

OpenRobotsTxt.org
Posts Replies Media Videos
openrobotstxt.org
So far we have seen almost 37,000 User-Agents mentioned in files on more than 600 million hostnames!

openrobotstxt.org
Stats from OpenRobotsTxt.org which says 600.5m hostnames scanned, 36,841 User-Agents found, and last updated on 15th May 2025.
May 15, 2025 at 11:15 AM