Frequently Asked Questions
Will the scan find every page?
No. Discovery depends on sitemaps, links, robots rules, access and task limits. Orphan pages may not be found.
Does the crawler respect robots.txt?
It reads robots.txt and skips disallowed paths. If robots cannot be read, the issue is reported; the crawler does not bypass site access controls.
Are imported URLs checked?
No. Imports validate URL structure, same-host consistency and duplicates, and are clearly marked as unchecked.
How long are tasks retained?
The default is 7 days, configurable with TASK_RETENTION_DAYS. Operators should verify cleanup after deployment.
Why might a scan fail?
Common causes include private DNS answers, robots restrictions, timeouts, redirect issues or resource limits. The crawler will not bypass access controls.