you know I thought adding a technobabbling AI-crawler honeypot to my server would be funny, but I didn't expect it to be that effective
was running out of RAM because openAI GPTBot is stuck in the infinite honeypot
(it's https://maurycyz.com/projects/trap_bots/)
was running out of RAM because openAI GPTBot is stuck in the infinite honeypot
(it's https://maurycyz.com/projects/trap_bots/)
November 17, 2025 at 1:35 PM
you know I thought adding a technobabbling AI-crawler honeypot to my server would be funny, but I didn't expect it to be that effective
was running out of RAM because openAI GPTBot is stuck in the infinite honeypot
(it's https://maurycyz.com/projects/trap_bots/)
was running out of RAM because openAI GPTBot is stuck in the infinite honeypot
(it's https://maurycyz.com/projects/trap_bots/)
#OpenAI ignores robots.txt:
🔗 koldfront.dk/openai_crawl...
📣 #ChatGPT, #GPTBot, #LLM, #OAI, #robots.txt
🔗 koldfront.dk/openai_crawl...
📣 #ChatGPT, #GPTBot, #LLM, #OAI, #robots.txt
OpenAI crawling websites
koldfront.dk
November 15, 2025 at 8:41 AM
#OpenAI ignores robots.txt:
🔗 koldfront.dk/openai_crawl...
📣 #ChatGPT, #GPTBot, #LLM, #OAI, #robots.txt
🔗 koldfront.dk/openai_crawl...
📣 #ChatGPT, #GPTBot, #LLM, #OAI, #robots.txt
"In the past year, Common Crawl’s CCBot has become the scraper most widely blocked by the top 1,000 websites, surpassing even OpenAI’s GPTBot, which collects content for ChatGPT" www.theatlantic.com/technology/2... via @theatlantic.com
The Nonprofit Doing the AI Industry’s Dirty Work
The web archive Common Crawl has been quietly funneling paywalled articles to AI companies—and lying to publishers about it.
www.theatlantic.com
November 4, 2025 at 4:57 PM
"In the past year, Common Crawl’s CCBot has become the scraper most widely blocked by the top 1,000 websites, surpassing even OpenAI’s GPTBot, which collects content for ChatGPT" www.theatlantic.com/technology/2... via @theatlantic.com
Ich glaub dieser Vergleich hinkt ein bissl, in Anbetracht eures robots.txt. Zumindest hab ich bei noch keinem Blumengeschäft ein Schild "Liebe Roboter, bitte nix fladern" gesehen.
November 4, 2025 at 10:43 AM
Ich glaub dieser Vergleich hinkt ein bissl, in Anbetracht eures robots.txt. Zumindest hab ich bei noch keinem Blumengeschäft ein Schild "Liebe Roboter, bitte nix fladern" gesehen.
[Étude] Presse française versus IA : qui bloque quel bot ?
Étude exclusive sur 1132 sites de presse français : 23% bloquent les robots d'IA. CCBot en tête, suivi de GPTBot. Découvrez les stratégies de blocage des éditeurs.
Étude exclusive sur 1132 sites de presse français : 23% bloquent les robots d'IA. CCBot en tête, suivi de GPTBot. Découvrez les stratégies de blocage des éditeurs.
[Étude] Presse française versus IA : qui bloque quel bot ?
Étude exclusive sur 1132 sites de presse français : 23% bloquent les robots d'IA. CCBot en tête, suivi de GPTBot. Découvrez les stratégies de blocage des éditeurs.
www.lvlup.fr
November 3, 2025 at 9:00 AM
[Étude] Presse française versus IA : qui bloque quel bot ?
Étude exclusive sur 1132 sites de presse français : 23% bloquent les robots d'IA. CCBot en tête, suivi de GPTBot. Découvrez les stratégies de blocage des éditeurs.
Étude exclusive sur 1132 sites de presse français : 23% bloquent les robots d'IA. CCBot en tête, suivi de GPTBot. Découvrez les stratégies de blocage des éditeurs.
Welp, it turns out my site was getting slowed down massively because of Ai bots combing it for content. GPTbot had accessed it 1.34k times just this morning!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
October 30, 2025 at 1:21 PM
Welp, it turns out my site was getting slowed down massively because of Ai bots combing it for content. GPTbot had accessed it 1.34k times just this morning!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
I can take solace in the fact that at least my art has made their image generation marginally worse. Suck it big tech!!
Ok actually, because robots.txt is optional and ignorable, here is our current, full AI bot blocking solution using the non-optional .htaccess mod_rewrite (see alt text)
![<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
# block AI bots
RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Ahrefsbot|AI2Bot|Ai2Bot-Dolma|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|AspiegelBot|aspiegel|AwarioRssBot|AwarioSmartBot|Brightbot\s1.0|Bytespider|CCBot|chatgpt|ChatGPT|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Crawlspace|DataForSeoBot|Datanyze|Diffbot|DotBot|DuckAssistBot|FacebookBot|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|ISSCyberRiskCrawler|Kangaroo\sBot|Kinza|OPPO\sA33|magpie-crawler|Meta-ExternalAgent|Meta-ExternalFetcher|OAI-SearchBot|omgili|omgilibot|Omgilibot|PanguBot|peer39_crawler|PerplexityBot|PetalBot|serpstatbot|spaziodati|Scrapy|SemrushBot|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade\sindexer\sbot|thesis-research-bot|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YandexBot|YouBot|zh-CN|zh_CN) [NC]
RewriteRule ^ – [F]
</IfModule>](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:eb2wskwyww57welmbn7c6mak/bafkreigdduympefft24ufhmoggotfu76lokjng5zt42khy4zvku75haj5i@jpeg)
October 28, 2025 at 5:09 PM
Ok actually, because robots.txt is optional and ignorable, here is our current, full AI bot blocking solution using the non-optional .htaccess mod_rewrite (see alt text)
Just adding a bunch of new/renamed AI scraper bots to the ol blocklist
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
October 28, 2025 at 1:41 PM
Just adding a bunch of new/renamed AI scraper bots to the ol blocklist
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: semantic-visions.com
Disallow: /
Am I Visible on AI : Vérifiez votre visibilité sur ChatGPT#Seo #ChatGPTcrawler #GPTBot #optimisationcontenuIA #robots.txtIA #visibilitéIA
October 24, 2025 at 7:57 AM
Am I Visible on AI : Vérifiez votre visibilité sur ChatGPT#Seo #ChatGPTcrawler #GPTBot #optimisationcontenuIA #robots.txtIA #visibilitéIA
とはいえ、GPTBot のクロールをブロックしてるサイトでは ChatGPT Atlas からでもコンテンツにアクセスできないので UA 文字列自体はあまり重要じゃなかった
October 22, 2025 at 5:37 AM
とはいえ、GPTBot のクロールをブロックしてるサイトでは ChatGPT Atlas からでもコンテンツにアクセスできないので UA 文字列自体はあまり重要じゃなかった
RIP robots.txt (1994-2025)
Died when AI companies discovered "please don't" isn't legally binding.
ClaudeBot has a 70,900 crawls per referral, GPTBot has a 1,700:1 and Perplexity CEO: "lol, not a legal framework tho"
Turns out we're all just unpaid contributors
www.heise.de/en/backgroun...
Died when AI companies discovered "please don't" isn't legally binding.
ClaudeBot has a 70,900 crawls per referral, GPTBot has a 1,700:1 and Perplexity CEO: "lol, not a legal framework tho"
Turns out we're all just unpaid contributors
www.heise.de/en/backgroun...
Obituary: Farewell to robots.txt (1994-2025)
The voluntary compliance protocol that civilized the internet has departed, bids Henning Fries farewell.
www.heise.de
October 20, 2025 at 5:47 PM
RIP robots.txt (1994-2025)
Died when AI companies discovered "please don't" isn't legally binding.
ClaudeBot has a 70,900 crawls per referral, GPTBot has a 1,700:1 and Perplexity CEO: "lol, not a legal framework tho"
Turns out we're all just unpaid contributors
www.heise.de/en/backgroun...
Died when AI companies discovered "please don't" isn't legally binding.
ClaudeBot has a 70,900 crawls per referral, GPTBot has a 1,700:1 and Perplexity CEO: "lol, not a legal framework tho"
Turns out we're all just unpaid contributors
www.heise.de/en/backgroun...
"OpenAI led the charge with its GPTBot, ChatGPT-User, and OAI-SearchBot – a trinity of violations that left robots.txt helplessly watching its directives being diligently ignored. The company publicly claimed compliance, while in June 2025, Cloudflare documented a devastating crawl-to-referral […]
Original post on mastodon.social
mastodon.social
October 20, 2025 at 8:00 AM
"OpenAI led the charge with its GPTBot, ChatGPT-User, and OAI-SearchBot – a trinity of violations that left robots.txt helplessly watching its directives being diligently ignored. The company publicly claimed compliance, while in June 2025, Cloudflare documented a devastating crawl-to-referral […]
> Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; + https:// openai.com/gptbot ) Thousands of times in my personal nginx logs this evening. Even thought they're all getting 403'd. Kinda funny really. (I am aware of Anubis et.al.)
Interest | Match | Feed
Interest | Match | Feed
Origin
bsd.network
October 16, 2025 at 11:30 AM
@awinkler@openbiblio.social @digiSberlin@openbiblio.social wmf hat doch einen eigenen GPTbot, der kann doch die sparql abfragen generieren.
October 15, 2025 at 11:44 AM
@awinkler@openbiblio.social @digiSberlin@openbiblio.social wmf hat doch einen eigenen GPTbot, der kann doch die sparql abfragen generieren.
On Website Technicals (2025-06) - Tech updates: Junited - Rigby to Buttersafe - GPTBot badness, captions, diversion delay, under-volt, X11 fossil. #junited2025 - https://www.earth.org.uk/note-on-site-technicals-97.html
On Website Technicals (2025-06)
Tech updates: Junited - Rigby to Buttersafe - GPTBot badness, captions, diversion delay, under-volt, X11 fossil. #Junited2025
www.earth.org.uk
October 14, 2025 at 9:53 AM
On Website Technicals (2025-06) - Tech updates: Junited - Rigby to Buttersafe - GPTBot badness, captions, diversion delay, under-volt, X11 fossil. #junited2025 - https://www.earth.org.uk/note-on-site-technicals-97.html
more than 3,800 top domains are now blocking ai crawlers, and it’s not just openai's bots that are receiving the cold shouler.
cloudflare’s new data shows gptbot, ccbot, and google-extended lead the pack of most-disallowed ai agents.
cloudflare’s new data shows gptbot, ccbot, and google-extended lead the pack of most-disallowed ai agents.
October 10, 2025 at 7:01 PM
more than 3,800 top domains are now blocking ai crawlers, and it’s not just openai's bots that are receiving the cold shouler.
cloudflare’s new data shows gptbot, ccbot, and google-extended lead the pack of most-disallowed ai agents.
cloudflare’s new data shows gptbot, ccbot, and google-extended lead the pack of most-disallowed ai agents.
Google führt ein, dass Devoloper von Android-Apps sich identifizieren müssen
Alle Hersteller von LLM GPT führen Vibe Coding ein.
Wie funktioniert das? Der GPTbot identifiziert sich bei Google und verantwortet die App mit allen Modifikationen des menschlichen "Developers"?
Der menschliche […]
Alle Hersteller von LLM GPT führen Vibe Coding ein.
Wie funktioniert das? Der GPTbot identifiziert sich bei Google und verantwortet die App mit allen Modifikationen des menschlichen "Developers"?
Der menschliche […]
Original post on no-pony.farm
no-pony.farm
October 9, 2025 at 4:44 PM
Google führt ein, dass Devoloper von Android-Apps sich identifizieren müssen
Alle Hersteller von LLM GPT führen Vibe Coding ein.
Wie funktioniert das? Der GPTbot identifiziert sich bei Google und verantwortet die App mit allen Modifikationen des menschlichen "Developers"?
Der menschliche […]
Alle Hersteller von LLM GPT führen Vibe Coding ein.
Wie funktioniert das? Der GPTbot identifiziert sich bei Google und verantwortet die App mit allen Modifikationen des menschlichen "Developers"?
Der menschliche […]
From Googlebot to GPTBot: Who’s crawling your site in 2025 https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
From Googlebot to GPTBot: Who’s crawling your site in 2025
blog.cloudflare.com
October 6, 2025 at 8:24 AM
From Googlebot to GPTBot: Who’s crawling your site in 2025 https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
От Googlebot до GPTBot: кто будет сканировать ваш сайт в 2025 году
https://kripta.biz/posts/48494BB0-8029-4BEB-9CEE-7AF5EED9C6F8
https://kripta.biz/posts/48494BB0-8029-4BEB-9CEE-7AF5EED9C6F8
October 2, 2025 at 3:41 AM
От Googlebot до GPTBot: кто будет сканировать ваш сайт в 2025 году
https://kripta.biz/posts/48494BB0-8029-4BEB-9CEE-7AF5EED9C6F8
https://kripta.biz/posts/48494BB0-8029-4BEB-9CEE-7AF5EED9C6F8
October 2, 2025 at 3:40 AM
AI crawlers aren’t welcome everywhere.
Of the top 10,000 domains, 3,710 disallow them in robots.txt:
GPTBot leads with 240 full blocks + 66 partial
ClaudeBot: 161 full, 47 partial
PerplexityBot: 128 full, 52 partial
Even ChatGPT-User, Anthropic, and Amazonbot are on the block list
Of the top 10,000 domains, 3,710 disallow them in robots.txt:
GPTBot leads with 240 full blocks + 66 partial
ClaudeBot: 161 full, 47 partial
PerplexityBot: 128 full, 52 partial
Even ChatGPT-User, Anthropic, and Amazonbot are on the block list
September 9, 2025 at 5:38 AM
AI crawlers aren’t welcome everywhere.
Of the top 10,000 domains, 3,710 disallow them in robots.txt:
GPTBot leads with 240 full blocks + 66 partial
ClaudeBot: 161 full, 47 partial
PerplexityBot: 128 full, 52 partial
Even ChatGPT-User, Anthropic, and Amazonbot are on the block list
Of the top 10,000 domains, 3,710 disallow them in robots.txt:
GPTBot leads with 240 full blocks + 66 partial
ClaudeBot: 161 full, 47 partial
PerplexityBot: 128 full, 52 partial
Even ChatGPT-User, Anthropic, and Amazonbot are on the block list
gptbotがアクセスしてきた形跡があるんだけど気持ち悪い
September 7, 2025 at 1:04 AM
gptbotがアクセスしてきた形跡があるんだけど気持ち悪い
was scrolling through my Nginx logs earlier to debug something and noticed a lot of requests from GPTBot, so I ran a simple python script to analyse it.
September 4, 2025 at 11:48 PM
was scrolling through my Nginx logs earlier to debug something and noticed a lot of requests from GPTBot, so I ran a simple python script to analyse it.