Content Architecture Gap Checker methodology

You provide a same-site start URL (usually the homepage). Optionally add a competitor URL for high-level structure themes only never rank tracking. All fetches are public HTML/text with SSRF hardening.

• Fetch the start URL and extract same-host internal links
• Read robots.txt for Sitemap: lines when present
• Parse sitemap / sitemap index XML (public text fetch, SSRF-hardened)
• Prioritize taxonomy-looking paths, then BFS more internal URLs
• Hard cap: about 36 HTML pages and 200 sitemap URLs per run
• Optional: competitor homepage + sitemap themes (not rank tracking)

The budget is intentionally capped. Treat results as an architecture sample with prioritized seeds, not a complete site mirror.

• Weighted taxonomy points: dedicated = full, weak = partial, missing = zero
• Thin crawls are capped (single-page HTML cannot earn an exceptional grade)
• Evidence coverage reflects pages fetched vs budget and sitemap breadth
• Recommendations suggest concrete paths/titles for missing or weak areas

Example: a B2B SaaS site scores mid-band. Service and blog pages are dedicated; case studies and comparison routes are missing; pricing is weak (only a nav label, no fetched page in budget). Evidence coverage is moderate because the sitemap listed hundreds of URLs but only a capped subset was fetched.

• Ship missing conversion and proof types first when the goal is pipeline not more blog volume.
• Weak ≠ absent forever: a path signal without a dedicated page is a cue to verify CMS routes outside the crawl budget.
• Do not treat this as keyword gaps: missing “alternatives” pages is an architecture finding, not proof of search demand.