robots.txt for a Small Business Site: What to Allow, Block and Test
Short answer: A robots.txt file is a plain text file at the root of your domain that asks search crawlers to skip certain paths. For most small sites it needs only a few lines: a user-agent rule, a short list of private paths such as the admin area and the cart, and a link to your sitemap. The three mistakes that cause real damage are leftover staging rules that block the whole site, blocking the CSS and JavaScript Google needs to render pages, and trying to hide pages with robots.txt instead of a noindex tag. Test the file in Search Console before and after every change.
Where the file lives and what it does
The file must sit at the top level of each host, so https://www.example.com.au/robots.txt is read for that host only. A blog on a subdomain needs its own file. The file is public, and anyone can open it, so do not list sensitive folder names there that you would rather keep quiet.
Robots.txt controls crawling, not indexing. It tells Google which URLs it may fetch. A blocked page can still appear in results if other sites link to it, although Google will not know what is on it. To keep a page out of results, use a noindex instruction on the page itself, and make sure crawlers can reach that page so they can see the instruction.
A safe starter file
Here is a simple file for a small service business. The paths are examples, so swap them for the folders your site really uses:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://www.example.com.au/sitemap.xml
Each line does one job. The first line applies the rules to all crawlers. The three Disallow lines skip areas that have no value in search. The sitemap line points crawlers to the list of pages you do want found. Keep the file short, and leave out anything you would only add out of habit.
Mistake 1: a staging rule goes live
Developers often build a new site on a test address and block all crawlers there with two lines:
User-agent: *
Disallow: /
When that file is copied to the live site during a launch, Google is told to stay away from every page. Rankings and new indexing stop. Check the live file on the day you launch, and again a week later, because some deployment tools copy the file back from the test site.
Mistake 2: blocking CSS and JavaScript
Older guides suggested blocking folders such as /wp-content/ or /assets/. Google needs those files to render your pages the way visitors see them. If the styles and scripts are blocked, Google may misread the layout, the navigation or the content. Only block folders that hold private files, and never block the folders that hold stylesheets and scripts.
Mistake 3: hiding a page with robots.txt
A Disallow rule stops crawling, but it does not remove a page that is already in the index. A page you want hidden needs a noindex instruction. Add it in the page head or send it as a response header, and leave the page crawlable so Google can read the instruction. Blocking the page in robots.txt at the same time prevents Google from seeing the noindex, which means the page can stay listed.
How to test the file
- Open the file in a browser. Visit the root address and check that it shows the content you expect, with no error page.
- Use the robots.txt report in Search Console. Under Settings, look for the robots.txt report. It shows the file Google fetched, when it last fetched it and any errors or warnings.
- Run URL Inspection on key pages. Paste the address of a service page or a product page. If it says it is blocked by robots.txt, the rule is too broad.
- Compare the sitemap. Every page in your sitemap should be allowed. A page listed in the sitemap but blocked in robots.txt sends mixed signals.
- Repeat after every change. Keep a note of the date and the change so that you can match any drop in traffic to an edit.
A quick checklist before you publish
- The file is named exactly
robots.txtand sits at the root of the domain - No line says Disallow: / unless the site is meant to be hidden
- Stylesheets and scripts are not blocked
- The sitemap line uses the full address with https
- Private pages that must stay out of results carry a noindex instruction
- The Search Console report shows the latest version without errors
Reading the file as a crawler does
Crawlers read the file from the top, and they follow the group that best matches their name. A group that begins with User-agent: * applies to every crawler that has no group of its own. If you add a second group for a specific crawler, that crawler will follow only its group, so repeat any rule it must still obey. Forgetting this is a common reason a sitemap line or a folder rule seems to be ignored for one crawler.
Keep the file in plain text, with one directive per line and no HTML. A blank line separates groups. Comments can start with a hash sign, but keep them few, because the file is read by machines first and by people second.
FAQ
Should I block AI crawlers in robots.txt?
That is a policy choice about how your content is used, and it is separate from basic search setup. Decide it on purpose and write the rule in its own group, so it does not change how Google reads the rest of the file.
What if I already have a long robots.txt file?
Read it line by line and ask what each rule protects. Old rules for plugins you no longer use, or for folders that were removed, can usually go. Remove them in small batches so that you can see the effect in Search Console.
Will robots.txt speed up my site?
Only indirectly. Fewer pointless crawls can reduce load on a small server, but the file does not make pages load faster for visitors.
Can I use robots.txt to hide a private customer area?
Do not rely on it for privacy. Protect private areas with a login, and use robots.txt only to keep crawlers from wasting time on those paths.
If you would like us to review your file and check the live site after the next release, send us the address of your site.