How to v1.0, updated 2026-07-07

Reading server logs to see what crawlers do

Your access log is the ground truth of how bots actually crawl you.

Every request to your server is recorded in an access log: the URL, the time, the status code, and the user agent that asked. That log is the ground truth of how crawlers actually treat your site, not how you assume they do.

It answers questions nothing else can. Which bots visit, and how often. Which pages they crawl, and which they never touch. How much of their time goes on redirects, errors, and pages you do not care about. Whether a bot you want, like GPTBot, is reaching your key pages at all.

To read one, filter to the user agents you care about, then look at the status codes and the URLs. A lot of 404s means bots are chasing dead links. A lot of 301s means they are hopping through redirects. A pile of requests to parameter URLs means crawl budget is leaking there.

Topkay's log analyser takes an access log and does this for you: it groups by bot, counts the status codes, and surfaces the waste. It is the honest counterpart to the audit. The audit tells you what should happen. The log tells you what does.

Read next

What is ARMX

Agent Readiness and Machine eXperience: the five things every machine does with your site, and the one score that measures them.

What is answerability

How easily an AI answer engine can find and lift the answer from your page.

SEO, AEO and GEO, the difference

Ranking pages, being the answer, being cited by models. Three jobs, not three names.

All topics
Ask Topkay the analyst×
Ask me anything about your site. I read your twin and the Learn corpus, and cite both. Try a skill below.