Discover pages from links
Read sitemaps and follow internal links within the configured depth and robots policy.
SITE DISCOVERY · INDEX ANALYSIS · SITEMAP GENERATION
Make important pages
discoverable.
Scan publicly accessible pages, review HTTP status, canonical and noindex signals, then generate an XML sitemap.
Default cap: 1,000 URLs (operator-configurable). Imported URLs are not checked for HTTP status.
Scan task
Processed0
Indexable0
Errors0
Discovered0
Queued
Page results
| URL | HTTP | Indexing | Canonical |
|---|
Scan scope
Only pages allowed by robots.txt, discoverable within the configured depth and publicly reachable are included. A scan cannot guarantee discovery of every site URL. lastmod is included only when supplied by a trusted source.
SCAN INTELLIGENCE
Site discovery overview
Scan scopePublic pages
OutputStandard XML
RecoveryShareable task link
Review index signals
Inspect HTTP status, redirects, canonical, noindex and crawl errors.
Generate a sitemap
Export standard XML, split large results and download the package as ZIP.
From domain to sitemap
Submit a site, review the discovered results and download the files. A limited scan is not a complete inventory of every URL.
FOR PRACTICAL SEARCH WORKFLOWS
From pre-launch checks to ongoing content reviews
Build a reviewable URL inventory around public page discovery and index signals.
Before a site launch
Check whether public pages can be discovered through internal links or an existing sitemap, and spot redirects or inaccessible pages.
Content and technical reviews
Filter results by HTTP status, canonical and noindex signals when reviewing which pages belong in a sitemap.
Project handoff and review
Return to a report with its task link, then share the generated XML and ZIP files with the site team.
UNDERSTAND THE SCOPE
Common questions
Can the crawler find every page on a site?
No. It can only discover public pages allowed by robots.txt, within task depth and URL limits, that are reachable from a sitemap or internal links.
Does importing a TXT or CSV list check page status?
No. Import validates and normalizes URLs, then generates a sitemap. Import mode does not request those pages or verify their HTTP status.
How is lastmod determined?
It is included only when a sitemap or page source provides a trustworthy modification time. Scan time is never used as a fabricated update date.