A web spider, also called a web crawler or simply a spider, is a tool that automatically explores a website by following its links, forms and resources to build a map of every URL it can reach. In security testing, tools such as OWASP ZAP use a spider to discover pages before scanning them for vulnerabilities.
Why it matters: A security scanner can only test the pages it knows about. Functional test suites rarely visit every URL in an application, so forgotten admin pages, old endpoints or rarely used flows can go unscanned. Spidering widens coverage and often reveals parts of a system the team did not realise were still exposed.
In practice: The spider starts from a target URL and follows links, sources and anchors found in HTML, JavaScript and CSS, adding each one to a site tree. Settings such as maximum depth and maximum number of child pages limit how far it goes. A traditional spider parses static content, while an AJAX spider drives a real browser to explore JavaScript-heavy applications. The spider itself does not test anything: the resulting tree is handed to passive and active scanners.
Examples:
- Running a spider after an automated regression suite to find pages the tests never opened, before scanning them.
- Using an AJAX spider on a single-page application where links are generated by JavaScript.
- Limiting crawl depth on a large content site to keep the scan within the time available in a CI/CD pipeline.
Common pitfalls:
- Letting a spider loose on a site with links that delete or change data, such as "remove" buttons, which can damage test environments.
- Expecting a spider to reach pages behind logins or complex forms without configuring authentication or seeding it with known URLs.
Is a web spider the same as a scanner? No. A spider discovers what exists by crawling. A scanner then examines what was found for vulnerabilities, and the two are usually run one after the other.
