OpenRobotsTXT
@openrobotstxt.org
OpenRobotsTXT is an open archive of the world’s robots.txt files.
OpenRobotsTxt.org
OpenRobotsTxt.org
We’re excited to announce a new update to openrobotstxt.org!
You can now search the archive for individual websites’ robots.txt files and look up specific bot names.
blog.majestic.com/company/new-...
You can now search the archive for individual websites’ robots.txt files and look up specific bot names.
blog.majestic.com/company/new-...
New: Site Search and Bot Lookup for OpenRobotsTXT
Back in May, we launched OpenRobotsTXT, a project to archive and analyse the world’s robots.txt
blog.majestic.com
August 14, 2025 at 9:46 AM
We’re excited to announce a new update to openrobotstxt.org!
You can now search the archive for individual websites’ robots.txt files and look up specific bot names.
blog.majestic.com/company/new-...
You can now search the archive for individual websites’ robots.txt files and look up specific bot names.
blog.majestic.com/company/new-...
So far we have seen almost 37,000 User-Agents mentioned in files on more than 600 million hostnames!
openrobotstxt.org
openrobotstxt.org
May 15, 2025 at 11:15 AM
So far we have seen almost 37,000 User-Agents mentioned in files on more than 600 million hostnames!
openrobotstxt.org
openrobotstxt.org
We're delighted to announce OpenRobotsTxt.org - a project to archive and analyse the world’s robots.txt files, kicking off with analysis of 595 million hostnames courtesy of a huge export of data from @majestic.com.
OpenRobotsTXT
OpenRobotsTxt.org
May 15, 2025 at 9:10 AM
We're delighted to announce OpenRobotsTxt.org - a project to archive and analyse the world’s robots.txt files, kicking off with analysis of 595 million hostnames courtesy of a huge export of data from @majestic.com.