Concept v1.0, updated 2026-07-07

What crawl budget is

The finite attention a crawler gives your site, and how it gets wasted.

Crawl budget is the finite amount of fetching a crawler will do on your site in a given period. It is not unlimited. If a bot spends its budget on the wrong URLs it never reaches the ones that matter.

Budget is wasted on redirects, error pages, pages set to noindex, and duplicate URLs. A single page reachable at four addresses, with and without a trailing slash, with a tracking parameter, over HTTP and HTTPS, is four fetches for one page.

You trim it by keeping internal links pointing at the final canonical URL, stripping tracking parameters, resolving redirect chains, and aligning robots.txt with your sitemap so you are not listing URLs you also block.

Topkay finds this waste in the site audit and the log analyser: the redirects, the errors, and the noindex pages a bot is spending time on.

Read next

What is ARMX

Agent Readiness and Machine eXperience: the five things every machine does with your site, and the one score that measures them.

What is answerability

How easily an AI answer engine can find and lift the answer from your page.

SEO, AEO and GEO, the difference

Ranking pages, being the answer, being cited by models. Three jobs, not three names.

All topics
Ask Topkay the analyst×
Ask me anything about your site. I read your twin and the Learn corpus, and cite both. Try a skill below.