Lemonhail
LightNews LightNews
banner
lemonhail.bsky.social
Lemonhail
@lemonhail.bsky.social
160 followers 300 following 73 posts
Potential lottery winner
Posts Replies Media Videos
lemonhail.bsky.social
Look, I'm an overweight guy in my 30s so I think I know a little bit more about gymnastics than some guy who's only won a bunch of Olympic medals in gymnastics.
November 27, 2025 at 6:10 PM
lemonhail.bsky.social
It'll go lower
November 17, 2025 at 2:32 PM
lemonhail.bsky.social
Is he OK?
November 17, 2025 at 1:52 AM
lemonhail.bsky.social
Yeah I agree
November 8, 2025 at 11:26 AM
lemonhail.bsky.social
Mountain of shit is about to turn into gold, says king of shit mountain
November 6, 2025 at 11:32 PM
Reposted by Lemonhail
damonberes.com
Common Crawl says it complies with removal requests—while telling us they are “a pain in the ass”—but also is not actually removing the data in question.
In July 2023, The New York Times sent a notice to Common Crawl asking for the removal of previously scraped Times content. (In their lawsuit against OpenAI, the Times noted that Common Crawl includes “at least 16 million unique records of content” from Times websites.) The nonprofit seemed amenable to the request. In November of that year, a Times spokesperson, Charlie Stadtlander, told Business Insider: “We simply asked that our content be removed, and were pleased that Common Crawl complied.” But as I explored Common Crawl’s archives, I found that many Times articles appear to still be present. When I mentioned this to the Times, Stadtlander told me: “Our understanding from them is that they have deleted the majority of the Times’s content, and continue to work on full removal.” The Danish Rights Alliance (DRA), an organization that represents publishers and other rights-holders in Denmark, told me about a similar interaction with Common Crawl. Thomas Heldrup, the organization’s head of content protection and enforcement, showed me a redacted email exchange with the nonprofit that began in July 2024, in which the DRA requested that its members’ content be removed from the archive. In December 2024, more than six months after the DRA had initially requested removal, Common Crawl’s attorney wrote: “I confirm that Common Crawl has initiated work to remove your members’ content from the data archive. Presently, approximately 50% of this content has been removed.” I spoke with other publishers who’d received similar messages from Common Crawl. One was told, after multiple follow-up emails, that removal was 50 percent, 70 percent, and then 80 percent complete. By writing code to browse the petabytes of data, I was able to see that large quantities of articles from the Times, the DRA, and these other publishers are still present in Common Crawl’s archives. Furthermore, the files are stored in a system that logs the modification times of every file. The foundation adds a new “crawl” to its archive every few weeks, each containing 1 billion to 4 billion webpages, and it has been publishing these regular installments since 2013. None of the content files in Common Crawl’s archives appears to have been modified since 2016, suggesting that no content has been removed in at least nine years. In our first conversation, Skrenta told me that removal requests are “a pain in the ass” but insisted that the foundation complies with them. In our second conversation, Skrenta was more forthcoming. He said that Common Crawl is “making an earnest effort” to remove content but that the file format in which Common Crawl stores its archives is meant “to be immutable. You can’t delete anything from it.” (He did not answer my question about where the 50, 70, and 80 percent removal figures come from.) Yet the nonprofit appears to be concealing this from visitors to its website, where a search function, the only nontechnical tool for seeing what’s in Common Crawl’s archives, returns misleading results for certain domains. A search for nytimes.com in any crawl from 2013 through 2022 shows a “no captures” result, when in fact there are articles from NYTimes.com in most of these crawls. I also discovered more than 1,000 other domains that produce this incorrect “no captures” result for at least several of the crawls, and most of these domains belong to publishers, including the BBC, Reuters, The New Yorker, Wired, the Financial Times, The Washington Post, and, yes, The Atlantic. According to my research and Common Crawl’s own disclosures, the companies behind each of these publications have sent legal requests to the nonprofit. At least one publisher I spoke with told me that it had used this search tool and concluded that its content had been removed from Common Crawl’s archives.
November 4, 2025 at 12:18 PM
lemonhail.bsky.social
Interesting how the terms
'grassroots group' and 'campaign group' are used in that snippet
November 1, 2025 at 10:24 AM