Hey @johnmu.com

I just discovered roughly 200K URLs w status 204 (no content) not blocked in robots.txt.

Would a status 204 waste crawl budget or not?

I guess the question is: What IS a crawl (in terms of crawl budget)?

Is it both request, response header and content OR just request + response?

Is Googlebot capable of precognition?

Gbot reports a bug that was introduced after Gbot crawl.

Does anyone (perhaps @johnmu.com) know why GSC lists a page as last crawled May 21 21:28 - w user declared canonical showing a bug, that was introduced in a midnight release between the 21. and the 22?

@screamingfrog.co.uk Quick question regarding scheduled crawls: Does SF use the computer clock for activating the scheduled crawls at the right time or does SF use some other global clock and an internal sync'ed clock?

.
:: Is Google showing Favouritism? ::

After years (and years) of complaints,
of people showing G examples of weak, bad, spammy, unhelpful, unsatisfactory content,
ranking on "brand sites" (particularly Large/Enterprise Publishers) ...

... G made the #SRA.

>>>

X: x.com/darth_na/sta...

Hi @johnmu.com
Do you know how Gbot would interpret this in robots.txt:

/oplevelser/*$

GSC says it's crawlable when inspecting: www.dailys.dk/oplevelser/m...

technicalseo.com/tools/robots... + screaming Frog says its not due to robots.txt.

Is *$ an invalid combo making Gbot ignore that line?

I am seeing more big websites being hit by "bad URLs" in Google Search Console which indicates that the problem could be on a rise - perhaps leveraged by AI mass analyzing weak spots on big reputable websites. 😱💩

I wrote an article showing you how to fix it: www.linkedin.com/pulse/scamme...

@mortendd.bsky.social Fruen og jeg har lige været ved Børnnerup Havn i håb om at se hærfuglen (desværre var den væk).

MEN vi så dyrespor i sandet ved havnen og lugtede kraftigt dyretis 50 meter derfra.
Aftrykkene var på størrelse med 2/5-krone ca.
Kan du se, om sporene kan passe med Mårhund?