Lorixen.
EN

Is robots.txt blocking your CSS and JavaScript from Google?

Short answer: Google renders your pages much like a browser, so it needs to fetch the CSS, JavaScript and important images that build them. If your robots.txt disallows folders where those files live, Google may see a broken or empty page and understand your content less well. Google's own guidance is that you can block unimportant resources, but not ones whose absence makes the page harder to understand. Check what Google actually loads with the URL Inspection tool, then remove or narrow the rules that block essential files.

Why blocked resources matter

Google's JavaScript documentation describes three stages: crawling, rendering and indexing. Before Googlebot fetches any URL, it reads robots.txt and checks whether that URL is allowed. Pages are then rendered in a headless version of Chromium, which runs your JavaScript and applies your styles, and the rendered result is what Google uses to understand the page.

The same robots.txt check applies to every file the page needs. If a stylesheet, a script that loads your product list or the images in your main content are disallowed, the renderer simply doesn't get them. The page Google sees may be missing content, have layout that looks broken on mobile, or hide navigation links that only appear once scripts run.

Google's introduction to robots.txt puts it plainly: you can block resource files such as unimportant images, scripts or styles if pages won't be significantly affected, but if their absence makes the page harder for Google's crawler to understand, don't block them.

Rules that commonly cause trouble

Old robots.txt files often carry rules written to save crawl activity or hide "technical" folders. Typical patterns:

  • Disallow: /assets/ or /static/, where the site's CSS and JavaScript are built
  • Disallow: /wp-content/ or a themes or plugins folder on a CMS site
  • Disallow: /js/ or /css/ copied from an old template
  • Disallow: /*.js$ or /*.css$ using wildcards
  • Disallow: /api/ when the page loads its main content from that API in the browser
  • a blanket Disallow: / left over from a staging site, with a few Allow lines that miss the asset folders

Google supports two wildcards in paths: * matches any sequence of characters and $ marks the end of the URL. A rule like /*.js$ therefore blocks every URL ending in .js across the whole site.

Finding out what Google can't load

Use the URL Inspection tool in Search Console on a few important page types: the home page, a service or product page, and a blog post.

  1. Inspect the URL and run a live test.
  2. Look at the screenshot of the rendered page. Missing styling, empty sections or a page that looks like plain text are warning signs.
  3. Open View tested page, then More info, to see the list of page resources and whether any couldn't be loaded.

Resources that failed because of robots.txt are the ones to fix first. Some failures will be third-party files on other domains, such as a chat widget or analytics script; you can't change another site's robots.txt, and most of those won't affect how Google understands your content.

Fixing the rules safely

Start by deciding which files the page genuinely needs to show its main content and layout. Then:

  • Remove blanket asset blocks. Deleting Disallow: /assets/ or similar is often the simplest fix.
  • Add targeted Allow rules if needed. If a folder must stay mostly blocked, allow the specific subfolders or file types pages depend on.
  • Check API paths. If your pages fetch their main content from an API in the browser, Google's renderer needs to reach those endpoints too. Block only API paths that pages never use for visible content.
  • Look at user-agent groups. Google uses the group for the most specific user agent that matches its crawler and ignores the others, and the order of groups in the file doesn't matter. A permissive User-agent: * group won't help if a separate User-agent: Googlebot group still blocks assets.

Keep the file short and commented so the next person understands why each rule exists.

What robots.txt should and shouldn't do

Robots.txt controls crawling, not indexing. It's useful for keeping crawlers out of things like internal search results, filtered URL combinations or admin screens. It's the wrong tool for hiding a page from search results, and it's an unhelpful tool for "protecting" files the public site already sends to every visitor's browser. Anyone visiting the page downloads your CSS and JavaScript anyway.

Checking the fix

After you publish the updated robots.txt:

  1. Fetch /robots.txt in a browser to confirm the live file is the new one, not a cached or CDN copy.
  2. Rerun the URL Inspection live test on the same pages and compare the screenshot and resource list.
  3. Check pages built with different templates, because each may load different files.
  4. Watch Search Console over the following weeks for changes in indexed pages or enhancement reports on pages that previously rendered badly.

An illustrative example

A dental clinic in Tampa rebuilt its website, and the developer kept the old robots.txt, which disallowed /assets/ and /api/. The new site loads its treatment price list and booking slots from /api/ in the browser. URL Inspection shows treatment pages rendered without styling and with empty price sections. The team removes the asset block, allows the two API endpoints the pages call, keeps the admin area blocked, and reruns the live test. The rendered screenshot now matches what patients see.

FAQ

Will blocking CSS and JavaScript get my site penalised?

Google's documentation frames it as an understanding problem: if essential resources are blocked, Google may not analyse the page well, which can affect how it appears.

Should I block my images folder?

Only if those images don't matter for understanding pages and you don't want them in image search. Product and content images usually should stay crawlable.

How quickly does Google notice a robots.txt change?

It rereads robots.txt periodically. Live tests in URL Inspection use the current file, so you can check immediately.

Does this affect other search engines?

Most major crawlers follow robots.txt and many render pages, so the same fix generally helps them too.

If you'd like your robots.txt checked against what your pages actually need, send us your site's address.

Sources

WhatsAppBook a call