Lorixen.
EN

Reading the Crawl Stats report in Search Console

Short answer: The Crawl Stats report shows how often Google requested files from your site, what your server answered and whether Google ran into availability problems. Google itself says a site with fewer than a thousand pages usually does not need it. For a small business, the useful parts are host status (is anything stopping Google from reaching the site?), the response table (are there unexpected errors or redirects?) and the discovery crawls after you publish new pages. The rest is detail for larger sites.

Many Australian business owners find this report by accident, buried under Settings in Search Console, and worry about a spiky chart they do not understand. This explainer walks through each part in the order it appears, says what it means in plain terms, and flags the few situations where you should act.

Where to find it, and who it is for

You reach the report through Settings → Crawl stats in Search Console. Google's help page describes it as aimed at advanced users and says that if your site has fewer than a thousand pages, you should not need to use it or worry about this level of crawling detail. That is worth keeping in mind before you spend an afternoon on it.

It is also only available for root-level properties: a Domain property such as example.com.au, or a URL-prefix property at the root, such as https://example.com.au. If you only verified a subfolder, you will not see it.

How the counting works

Two details explain most "why does this look strange?" moments:

  • Requests, not pages. The report counts the actual URLs Google requested, and it does not merge them into canonical pages as some other reports do. If page A redirects to page B, which redirects to page C, that is three requests.
  • Resources count too. Images, CSS and JavaScript files that Google loads to render your pages appear as their own requests. A site with 40 pages can easily show hundreds of requests a day.

The data also covers both http and https for a URL-prefix property, and a Domain property shows requests across its subdomains. Google notes a known issue: some requests may not be counted, so small differences from your own server logs are expected.

The summary chart: requests, download size, response time

At the top are three lines:

  • Total crawl requests: every request in the period, successful or not.
  • Total download size: bytes Google downloaded. Shared resources that Google cached are only requested the first time.
  • Average response time: how long your server took per resource, averaged.

For a small site, ups and downs in requests are normal; a site redesign, a new sitemap or a batch of new pages can all cause a bump. What deserves attention is average response time climbing and staying high, because a slow server makes everything slower for visitors as well.

Host status: the part worth checking first

Host status tells you whether Google hit availability problems in three areas: fetching robots.txt, DNS resolution and server connectivity. The help page describes three states: no significant issue in the past 90 days; at least one significant issue in the last 90 days but more than a week ago; or one in the last week.

Click into any category to see a chart with a dotted red line. If the metric crosses that line on a given day (Google gives the example of DNS failing for more than 5% of requests), that day counts as an issue.

The robots.txt detail matters more than most owners realise. Your site does not need a robots.txt file, but the request for it has to get a sensible answer. According to Google, a 200 with a file (even an empty or invalid one) counts as successful, and so does 403, 404 or 410, which simply means "no file". A 429 or 5xx response counts as unsuccessful. If Google cannot get an acceptable answer, it stops crawling for the first 12 hours, then uses the last good copy for up to 30 days, and after that behaves depending on whether your homepage is reachable; Google's robots.txt specification describes the same sequence and adds that 4xx codes other than 429 do not affect crawl rate. A misconfigured firewall or security plugin that blocks robots.txt can quietly slow crawling across the whole site.

Crawl responses: what the codes are telling you

This table shows responses as a share of all requests. Google groups them roughly like this:

  • Good: 200, 301/308 permanent redirects, 302/307 temporary redirects, meta refresh and 304 "not modified".
  • Possibly fine: 404. Google says it is not possible, worthwhile or even desirable to fix every 404; often it is the correct answer for a page that is truly gone.
  • Worth fixing: robots.txt unavailable, 401/407 unauthorised, 5xx server errors, other 4xx errors, DNS problems, timeouts and redirect errors such as loops.

The quick check for a small business: is the share of redirects suddenly large (an old link structure still being followed, or a redirect chain), and are any server errors appearing on days you can match to an outage or a plugin update?

File type, purpose and Googlebot type

  • File type shows what Google requested: HTML, images, CSS, JavaScript, PDFs and more. Google suggests looking here if response times are slow, for example to see whether many small images or resources on a slower external host are involved.
  • Crawl purpose splits requests into Discovery (a URL Google had never crawled) and Refresh (a recrawl of a known page). After you add a batch of new pages or submit a sitemap, you would ideally see a bump in discovery crawls. If discovery stays flat for weeks after a launch, check that the new pages are linked from your menus and listed in your sitemap.
  • Googlebot type shows which crawler made the request: smartphone, desktop, image, video, page resource load, AdsBot and others. Most small sites see smartphone dominate.

Clicking any row opens example URLs. Google is clear that these are representative samples, not a complete list.

When to act, and when to close the tab

Act if host status shows a recent problem, if server errors appear repeatedly, if robots.txt is reported as unavailable, or if response time is rising steadily. Close the tab if your site is small, host status is clean and the only thing bothering you is that requests went up one week and down the next.

FAQ

Why are there far more crawl requests than pages on my site?

Each image, stylesheet, script and redirect step counts as a request, and Google recrawls known pages. That's normal.

Does a drop in crawl requests mean my rankings will fall?

Not by itself. Check host status and response codes first; if they're clean, a quieter week usually means nothing.

My robots.txt shows as unavailable, but I don't have one. Is that a problem?

Having no file is fine if your server answers with a 404 or similar. "Unavailable" usually means the server returned an error or timed out for that request, which is worth fixing with your host.

Should I ask Google to crawl less?

Only if crawling is genuinely straining your server. Google's report links to a way to request a lower crawl rate, but small sites rarely need it.

When host status turns amber and you are not sure whether it is your server, your DNS or a plugin, send us a screenshot of the report and we will help you read it.

Sources

WhatsAppLet’s talk