Lorixen.
EN

Search Console "Unauthorized (401)" and "Forbidden (403)" Pages: Who Is Being Blocked?

Short answer: A 401 means the server asked Googlebot to log in; a 403 means it refused outright. For pages meant to be private, such as account areas or staging, that is exactly right and you can leave them. For public pages, it usually points to a firewall, security plugin, bot-protection rule or hosting setting that treats Googlebot as a threat. Check which URLs are listed, test them with the URL Inspection live test, then adjust the rule that is turning Googlebot away rather than disabling protection altogether.

Read the two statuses correctly

The Page indexing report in Search Console lists pages Google could not index and why. Two of the reasons concern access:

  • Blocked due to unauthorized request (401): the server replied that authentication is required.
  • Blocked due to access forbidden (403): the server understood the request but declined to serve it, even though no login prompt was offered.

Neither status says anything about your content. They say what the server did when Google knocked. First, check whether the listed URLs should be public at all.

Step one: sort the URLs into two piles

Export the list from the report and divide it.

Pile Examples Action
Should be private /my-account/, /wp-admin/, staging subdomain, members area, client portal Nothing. A 401 or 403 is the right answer. Consider noindex for tidiness
Should be public Service pages, blog posts, product pages, image files, CSS and JS Investigate, because Google cannot see them

If every listed address is private, the report is simply recording your protection working. Mark it as understood and move on.

Step two: find out what is refusing Googlebot

Public pages that return 401 or 403 to Google typically have one of these causes.

  1. Web application firewall rules. Hosting-level or plugin firewalls sometimes challenge or block automated visitors by pattern, rate or country. A crawler fetching many pages quickly can trip a rate limit.
  2. Bot protection and challenge pages. A "verify you are human" screen cannot be solved by a crawler, so it records a refusal.
  3. Password protection left on. A site-wide HTTP basic authentication rule, applied during development, may still be active on the live domain or on a folder.
  4. Geo-blocking. Googlebot crawls mostly from US addresses. If you block that country to reduce spam, Google is turned away too.
  5. Hotlink or directory rules. A rule meant to stop others embedding your images may also refuse image requests that arrive without a referrer.
  6. Security plugin lists. Some block user agents they do not recognise, or addresses from cloud providers.
  7. CDN settings. A CDN may apply its own block or challenge before your server sees the request.

Step three: reproduce what Google sees

Do not guess. Use the tools to see the same response.

  1. In Search Console, run URL Inspection on one affected public address and choose "Test live URL". If the live test fetches the page successfully, the block may be intermittent, tied to load or time of day.
  2. Check your host's firewall or security log for the time of the crawl and look for the Google user agent or its addresses. Hosting dashboards and many security plugins keep a blocked-request log.
  3. Compare with the Crawl Stats report under Settings, which breaks requests down by response code and shows whether 4xx responses are rising.
  4. Make sure you are not misled by impostors. Anyone can send a request claiming to be Googlebot, so verify with a reverse DNS lookup or by comparing against Google's published address ranges before allowing anything by user agent alone.

Step four: fix the rule, not the whole defence

  • In a firewall or CDN, add an exception for verified Googlebot traffic, or lower the strictness of the challenge for crawlers, rather than turning the product off.
  • Remove basic authentication from the live site and keep it on staging only.
  • Reduce rate limits on static assets and public pages so a normal crawl does not trigger them.
  • Review geo rules and exempt crawlers from them.
  • Where a plugin offers an allow-list for search engines, enable it.

After changing a rule, retest the live URL, then use "Validate fix" if Search Console offers it. Expect a few days to a couple of weeks for the status to clear.

Illustrative scenario

A US landscaping firm installs a security plugin with strict rate limiting. A week later Search Console shows forty public pages under 403 and traffic dips. The log shows the plugin challenging rapid requests from one address range. The firm adds an allow rule for verified search crawlers, retests the live URL, and the following crawl returns 200 responses.

What not to do

  • Do not disable all protection to quiet the report.
  • Do not allow anyone claiming to be Googlebot without verification, since scrapers copy that name.
  • Do not rely on robots.txt for pages that must stay private; use real authentication.
  • Do not remove a 401 from a members area just to tidy the report. Hiding private pages from the report is not worth exposing them.

FAQ

Do 401 and 403 pages hurt the rest of my site's rankings?

Not on their own. They only matter if public pages are refused, because those cannot be indexed.

Why does the live test pass while the report still shows errors?

The report reflects past crawls. If the issue was intermittent or has been fixed, the status clears after Google recrawls and validates.

Should private pages be noindexed?

It does not hurt, and it keeps them out of results if someone links to them. Real access control remains the protection.

Can a 403 appear only for images or CSS?

Yes. Hotlink or firewall rules often affect assets. Blocked stylesheets and scripts may stop Google rendering the page properly.

If your security settings and search visibility seem to be pulling against each other, reach out through our enquiry page.

Sources

WhatsAppBook a call