You can find the list on my blog (talkingtoavoidcalledtheinternet.wordpress.com/the-ultimate...) or you can download it directly from Google Drive - drive.google.com/file/d/1T3ix...

I have also added some helpful bots that you can optionally allow, as well as the ‘Crawl-delay’ command for every disallowed bot with a duration of 24 hours, or 86400 seconds.

In theory this should mean that banned bots won’t try to access your site more than once per day.

It’s quite telling to see which bots get blocked by the big sites like Facebook or Wikipedia, for example. In any case, this list was compiled from the robots.txt files of major websites and from other such lists.

I have collated this list by looking at the robots.txt of major websites, which anyone can do by going to the homepage of any website and adding ‘/robots.txt’ to the end. For example:

bsky.app/robots.txt

The big problem is that the robots.txt is a voluntary standard, meaning bots can ignore it. Thankfully a lot of bots still do respect robot.txt files, and it is better to have a robots.txt instead of nothing.

www.rfc-editor.org/rfc/rfc9309....

I'd hazard the guess that the advert is a YouTube video which got hit by a copyright strike for the music they tried to use.

Still, running an advert for a soundbar without any sound is peak irony.

Well at least 2 people knew what to get me for my birthday.

@jayforeman.bsky.social @markcooperjones.bsky.social

I also realise that by tagging them both, it implies Mark and Jay are the two people who bought them for me.