Lorixen.
EN

Server log analysis: see how Googlebot really crawls your site

Short answer: Your server's access log records every request Googlebot makes, URL by URL, which Search Console's Crawl Stats report only shows as totals and sample URLs. Ask your host for the raw access logs, filter for Googlebot, confirm the hits are genuine with a reverse and forward DNS check, then group the URLs: if a large share go to filter parameters, redirects or error pages, that's crawling you can reduce. For a site under about a thousand pages, Google itself says you shouldn't need this level of detail, so start with Crawl Stats and only open the logs when something looks wrong.

Logs versus Crawl Stats in one paragraph

Search Console's Crawl Stats report shows total crawl requests, download size, average response time, host status, and breakdowns by response, file type, purpose (discovery of new URLs versus refresh of known ones) and Googlebot type. Its example URLs are, in Google's words, not comprehensive, just a representative sample, and Google notes there may be slight differences against your own request logs. Your logs are the opposite: every single request, but raw, unlabelled, and including plenty of bots that only claim to be Google. The two work best together: Crawl Stats tells you something changed, the logs tell you which URLs.

A worked example: a hypothetical tea shop in Vienna

Imagine a small web shop selling loose-leaf tea, about 600 product and category pages, running on a typical shared host. The founder notices in Crawl Stats that total crawl requests roughly doubled over a month while new products weren't being picked up quickly. She downloads a week of access logs. A few lines, simplified:

66.249.66.1 - - [20/Oct/2026:03:14:07 +0200] "GET /tea/green?sort=price&page=7 HTTP/1.1" 200 18234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.66.1 - - [20/Oct/2026:03:14:09 +0200] "GET /tea/green?sort=name&colour=jade HTTP/1.1" 200 18102 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...)"
66.249.66.1 - - [20/Oct/2026:03:15:40 +0200] "GET /old-shop/sencha.html HTTP/1.1" 301 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...)"
66.249.66.1 - - [20/Oct/2026:03:15:41 +0200] "GET /tea/sencha HTTP/1.1" 301 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...)"
203.0.113.50 - - [20/Oct/2026:03:16:02 +0200] "GET /wp-login.php HTTP/1.1" 404 512 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; ...)"

Each line gives the IP address, timestamp, the request (method and URL), the status code, the response size and the user agent. Here's how she reads it.

Step 1: Is it really Google?

Anyone can put "Googlebot" in a user agent. Google's verification guide describes a manual check that's enough for most cases:

  1. Run a reverse DNS lookup on the IP with the host command.
  2. Check the resulting hostname ends in googlebot.com, google.com or googleusercontent.com.
  3. Run a forward lookup on that hostname and confirm it returns the original IP.

Google's own example uses 66.249.66.1, which resolves to crawl-66-249-66-1.googlebot.com and back again. The last line in our extract, from 203.0.113.50 asking for a WordPress login page on a shop that doesn't run WordPress, won't pass that test. It's a scanner in disguise and should be excluded from any crawl analysis. For bulk checks, Google publishes its crawler IP ranges as JSON files, which spreadsheet or script users can match against instead of looking up each address.

Step 2: Group the verified hits by URL pattern

After filtering, she exports Googlebot's requests to a spreadsheet and groups the paths. In our hypothetical week the pattern is clear: a big share of hits go to category URLs with sort=, colour= and page= parameters in endless combinations. The shop's filter links generate a new crawlable URL for every combination, and Googlebot is dutifully following them.

Google's crawl budget guide describes exactly this risk: if Google spends too much time crawling URLs it shouldn't, it might not explore the rest of your site. Its suggestions include consolidating duplicate content and blocking crawling of URLs you don't want crawled with robots.txt.

Step 3: Look at the status codes

Two lines show a redirect chain: the old shop URL /old-shop/sencha.html redirects to /tea/sencha, which redirects again (perhaps to a trailing-slash version). The crawl budget guide says to avoid long redirect chains because they have a negative effect on crawling. The fix is to point the oldest URL straight at the final one, and to update internal links and the sitemap so they use final URLs.

Step 4: Decide the fixes

For the hypothetical tea shop:

  • Make sort and filter combinations non-crawlable (robots.txt rules for the parameter patterns, and filter links that don't create new indexable URLs), while keeping normal paginated category pages reachable.
  • Collapse the redirect chains to single hops.
  • Check the sitemap lists only final, canonical product URLs with accurate lastmod dates.
  • Re-check Crawl Stats a few weeks later: discovery crawls should rise when new products are added, and refresh crawls should concentrate on real pages.

A routine for your own site

Getting logs. Most hosts provide raw access logs in the control panel, sometimes only for the last few days unless you enable archiving. Managed shop platforms often don't expose server logs at all; in that case, Crawl Stats is your only view, and that's fine. If a CDN sits in front of your site, the CDN's logs may be where the requests actually show up.

Tools. A spreadsheet is enough for a week of logs on a small site. Command-line tools (grep, awk) or a dedicated log analyser help once files get large. Whatever you use, filter verified Googlebot first and group by path pattern before looking at individual URLs.

What to look for, every time:

Pattern in the logs What it usually means Typical fix
Many hits on parameter URLs Filters or sorting creating endless URLs Robots.txt rules, fewer crawlable filter links
Repeated 301s, two hops or more Redirect chains from old migrations Point old URLs at the final destination
Regular hits on 404s from old URLs Links or sitemaps still referencing them Update links; leave true removals as 404
Spikes in 5xx responses Server overloaded or failing Talk to your host; check average response time in Crawl Stats
Important pages rarely requested Weak internal linking or missing from sitemap Link them from relevant pages, add to sitemap

When to repeat it. After a migration, after adding faceted navigation, when Crawl Stats shows a jump in requests or response time, or when new pages take noticeably long to appear in search.

When it's overkill

Google is direct about this. The Crawl Stats help says that if your site has fewer than a thousand pages you shouldn't need the report or this level of crawl detail, and the crawl budget guide is aimed at large sites (roughly a million pages) or medium sites with very rapidly changing content. A ten-page brochure site doesn't need log analysis. A shop with hundreds of products and filter navigation is borderline; it's worth one pass after launch and after big changes, not a weekly ritual.

FAQ

Can I see Googlebot's requests if my shop is on a hosted platform?

Often not as raw logs. Use Search Console's Crawl Stats report instead; it shows responses, file types and example URLs, which is enough for most small shops.

Why do my logs show more "Googlebot" hits than Search Console?

Partly because many fake bots use Google's user agent, and partly because Google says its report can differ slightly from request logs. Verify IPs before comparing numbers.

Should I block parameter URLs in robots.txt?

Only the ones that create duplicate or near-endless variations, like sort orders and filter combinations. Blocked URLs, as Google notes, stay in the crawl queue longer, so don't block pages you actually want indexed.

How much log data do I need?

For a small site, one to two weeks usually shows the pattern. For a shop with frequent product changes, a month gives a steadier picture.

When your filter pages are eating Google's attention, we can map the crawl waste and fix the shop's URLs with you.

Sources

WhatsAppLet’s talk