crwlr.software
@crwlr.bsky.social
crwlr.software is a collection of open source PHP composer packages for web crawling and scraping.
😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈
October 15, 2025 at 12:39 PM
😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈😈
Made it to 10,000 installs of the crwlr/crawler package! 🥳💪🏻
Here’s to the next 10,000! 🥂
Here’s to the next 10,000! 🥂
July 31, 2025 at 5:58 PM
Made it to 10,000 installs of the crwlr/crawler package! 🥳💪🏻
Here’s to the next 10,000! 🥂
Here’s to the next 10,000! 🥂
Reached some install milestones for the crwlr packages 🎉
- url passed 50,000 installs
- query-string is at an unbelievable 3,000,000 installs (thanks to bref!)
- robots-txt is approaching 15,000
- crawler will soon reach 10,000
Thanks to everyone using and supporting the packages! 🫶
- url passed 50,000 installs
- query-string is at an unbelievable 3,000,000 installs (thanks to bref!)
- robots-txt is approaching 15,000
- crawler will soon reach 10,000
Thanks to everyone using and supporting the packages! 🫶
May 7, 2025 at 8:09 AM
Reached some install milestones for the crwlr packages 🎉
- url passed 50,000 installs
- query-string is at an unbelievable 3,000,000 installs (thanks to bref!)
- robots-txt is approaching 15,000
- crawler will soon reach 10,000
Thanks to everyone using and supporting the packages! 🫶
- url passed 50,000 installs
- query-string is at an unbelievable 3,000,000 installs (thanks to bref!)
- robots-txt is approaching 15,000
- crawler will soon reach 10,000
Thanks to everyone using and supporting the packages! 🫶
Made it to 350 ⭐️s on the crawler package Github repo! 🚀💪🏻🫶
github.com/crwlrsoft/cr...
github.com/crwlrsoft/cr...
GitHub - crwlrsoft/crawler: Library for Rapid (Web) Crawler and Scraper Development
Library for Rapid (Web) Crawler and Scraper Development - crwlrsoft/crawler
github.com
January 28, 2025 at 9:41 AM
Made it to 350 ⭐️s on the crawler package Github repo! 🚀💪🏻🫶
github.com/crwlrsoft/cr...
github.com/crwlrsoft/cr...
Tagged a bugfix release of the robots-txt package, thx to this issue github.com/crwlrsoft/ro... by Martin-Matchory. 🙏💪🏻
Doesn't seem to work with this robots.txt · Issue #12 · crwlrsoft/robots-txt
robots.txt ---------------------------- User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: * Disallow: /*blackhole Disallow: /?blackhole ---------------------------- url...
github.com
January 27, 2025 at 6:09 PM
Tagged a bugfix release of the robots-txt package, thx to this issue github.com/crwlrsoft/ro... by Martin-Matchory. 🙏💪🏻
A new blog article is out! 🤓📚
Learn how to use a crawler to perform a health check and warm your cache after every deployment of your website or web app. Ensure smooth performance and error-free user experiences. Happy reading!
www.crwlr.software/blog/post-de...
Learn how to use a crawler to perform a health check and warm your cache after every deployment of your website or web app. Ensure smooth performance and error-free user experiences. Happy reading!
www.crwlr.software/blog/post-de...
Crwlr Recipes: Using a Crawler for Website Error Detection and Cache Warming
Have you ever deployed your website or web app, only to discover hours later that you’ve introduced bugs or broken links? Or do you clear the cache with every deploy, leaving the first users to experi...
www.crwlr.software
January 20, 2025 at 8:54 AM
A new blog article is out! 🤓📚
Learn how to use a crawler to perform a health check and warm your cache after every deployment of your website or web app. Ensure smooth performance and error-free user experiences. Happy reading!
www.crwlr.software/blog/post-de...
Learn how to use a crawler to perform a health check and warm your cache after every deployment of your website or web app. Ensure smooth performance and error-free user experiences. Happy reading!
www.crwlr.software/blog/post-de...
Ah, it seems like XPath is no longer an option in the new DOM API in PHP 8.4. I assume it won’t be a problem if we remove this feature in a new major version of the crwlr/crawler library?
December 5, 2024 at 12:42 PM
Ah, it seems like XPath is no longer an option in the new DOM API in PHP 8.4. I assume it won’t be a problem if we remove this feature in a new major version of the crwlr/crawler library?
The new HTML5/DOM functionality in PHP 8.4 is amazing! I’m working on adding it to crwlr/crawler to leverage it when PHP 8.4 is used. This will enable features like CSS pseudo-selectors not supported by Symfony DomCrawler.
December 3, 2024 at 11:47 AM
The new HTML5/DOM functionality in PHP 8.4 is amazing! I’m working on adding it to crwlr/crawler to leverage it when PHP 8.4 is used. This will enable features like CSS pseudo-selectors not supported by Symfony DomCrawler.
PHP 8.4 just dropped! 🥳🚀💪
php.net/releases/8.4...
Great release page! Well done!
Already added running tests on 8.4 in CI, so all crwlr packages should be ready for the upgrade! 💪🙂
php.net/releases/8.4...
Great release page! Well done!
Already added running tests on 8.4 in CI, so all crwlr packages should be ready for the upgrade! 💪🙂
PHP 8.4 Released
PHP 8.4 is a major update of the PHP language. It contains many new features, such as property hooks, asymmetric visibility, an updated DOM API, performance improvements, bug fixes, and general cleanu...
php.net
November 21, 2024 at 10:45 AM
PHP 8.4 just dropped! 🥳🚀💪
php.net/releases/8.4...
Great release page! Well done!
Already added running tests on 8.4 in CI, so all crwlr packages should be ready for the upgrade! 💪🙂
php.net/releases/8.4...
Great release page! Well done!
Already added running tests on 8.4 in CI, so all crwlr packages should be ready for the upgrade! 💪🙂
Hey Bluesky! 👋
What better time to move over from the old place? So, crwlr is now here on Bluesky too! 🥳🚀
What better time to move over from the old place? So, crwlr is now here on Bluesky too! 🥳🚀
November 21, 2024 at 10:45 AM
Hey Bluesky! 👋
What better time to move over from the old place? So, crwlr is now here on Bluesky too! 🥳🚀
What better time to move over from the old place? So, crwlr is now here on Bluesky too! 🥳🚀