Reading:
Web Crawlers
Share:
Try JetBrains Aqua – An IDE for Writing Tests You Can Be Proud of
JetBrains Aqua is an all-in-one polyglot IDE for test automation that supports Selenium, Cypress, and Playwright. Download Aqua and see how it can empower you to build better software!

Web Crawlers

Here's a handy list of web crawler tools for software testers that you might find useful.

  • Dead Link Checker — Free broken link checker.
  • error404 — Check your website for broken links.
  • Monkey Test —  Spider Monkey finds broken links.
  • Screaming Frog (SEO Spider) — The SEO Spider is a desktop program you can install locally on PC, Mac or Linux which crawls websites’ links, images, CSS, script and apps to evaluate onsite SEO.
  • Site Spider - Use this extension to spider a website looking for dead links.
  • W3C's Link Checker — Check links and anchors in Web pages or full websites.
  • Xenu Link Sleuth (download) — Find broken links on web sites.
  • 404-checker — 404 Checker is a website full of webmaster tools and information about your website, internet connections and protocols.

More to add?

Are there any resources we’ve missed? Do you know of any great web crawler tools we should add to this list? Share your findings on The Club and we’ll review it to be added to this list.

Try JetBrains Aqua – An IDE for Writing Tests You Can Be Proud of
JetBrains Aqua is an all-in-one polyglot IDE for test automation that supports Selenium, Cypress, and Playwright. Download Aqua and see how it can empower you to build better software!
Tips For The Lone Tester: Challenges With Software
🚨 BREAKING NEWS - The Explorer Discovered a New Planet, The Testing Planet
Feature Spotlight: Setting up Bindings in SpecFlow
Explore MoT
30 Days of AI in Testing
Upgrade Your Testing Game with the 30 Days of AI in Testing Challenge!
A Software Tester’s Guide To Chrome Devtools
Learn how to dig deeper into the Web with the use of Devtools
  • Viv Richards's profile