Tutorial v1.0, updated 2026-07-07

Trim crawl waste on your site

Stop bots spending their budget on redirects, errors and duplicates.

The outcome: crawlers spend their time on your real pages, not on waste. Less waste means more of your real content gets crawled, indexed, and answered from. See crawl budget.

  1. Run Site audit, or build your twin, to crawl the site and roll up the issues: redirects, errors, noindex pages, thin content, and near duplicates.
  2. Run the Log analyser on a server access log to see what bots are actually fetching, and how much of it is redirects and errors. This is what waste looks like in practice.
  3. Fix the big ones. Point internal links at final URLs so nothing redirects. Strip tracking parameters. Collapse redirect chains. Set a canonical on duplicates, and use the header form for non HTML ones.
  4. Check robots.txt against your sitemap with the Robots tester. A URL listed in the sitemap and blocked in robots is a contradiction that wastes budget.
  5. Re-crawl and confirm the issue counts drop.

Read next

What is ARMX

Agent Readiness and Machine eXperience: the five things every machine does with your site, and the one score that measures them.

What is answerability

How easily an AI answer engine can find and lift the answer from your page.

SEO, AEO and GEO, the difference

Ranking pages, being the answer, being cited by models. Three jobs, not three names.

All topics
Ask Topkay the analyst×
Ask me anything about your site. I read your twin and the Learn corpus, and cite both. Try a skill below.