Bad bots hammering your site: rate limits and firewall rules
Short answer: Bad bots ignore robots.txt, so stopping them happens at the network edge: a web application firewall with managed rules, rate limits on the URLs being abused, and challenges for suspicious traffic. First confirm which bots are real search crawlers so you don't block them, then match each symptom (slow server, fake accounts, scraping, login attempts) to a targeted rule. Check the logs after every change, because over-eager rules block customers too.
Your hosting dashboard shows CPU pinned at 100% at 3 a.m. Sign-ups jump overnight with names like "xjqkdlw". A competitor's site has your product descriptions word for word. None of these need a human attacker; they're the everyday work of automated traffic. The good news is that small sites can handle most of it with settings rather than custom code.
Good bots, bad bots and why robots.txt isn't the answer
Not all bots are a problem. Search engine crawlers, uptime monitors and link-preview fetchers are useful. Cloudflare, for example, maintains a list of verified bots: bots and agents it has confirmed identify themselves honestly and don't abuse their access, with categories such as search engine crawlers (Googlebot and Bingbot among the examples) and link-preview tools.
The trouble is bots that lie about who they are. Google's robots.txt introduction is clear that robots.txt instructions can't enforce anything: respectable crawlers obey them, other crawlers might not. A scraper doesn't read your robots.txt politely and leave. Blocking has to happen where you can actually refuse requests.
Check before you block "Googlebot"
Anyone can put "Googlebot" in a user-agent string. Google's verification guide describes two methods: a reverse DNS lookup on the IP address, checking that it resolves to googlebot.com, google.com or googleusercontent.com, followed by a forward lookup to confirm it matches; or, at scale, matching IPs against the published lists of Google IP ranges. Many firewalls do this for you through a "verified bot" setting. Never block by user agent alone in either direction.
Symptom by symptom
The server is slow or crashing at odd hours
Likely cause: aggressive crawling of many pages at once, often search pages, filtered category pages or calendar pages that generate endless URL combinations.
Fix:
- Add a rate limit on the heavy paths. Cloudflare's rate limiting rules are defined by an expression matching the traffic, a period in seconds, a number of requests per period, an action and a mitigation duration once the limit is hit.
- Cache pages that don't change per visitor, so repeat requests never reach your server.
- Look in the logs for the top IP ranges and paths; a single network hammering
/search?q=is a strong hint.
Fake accounts and sign-ups
Likely cause: scripts submitting the registration form directly.
Fix:
- Rate-limit the sign-up endpoint per IP.
- Add a challenge on that one page rather than the whole site.
- Require email confirmation before an account can do anything visible.
- Delete unconfirmed accounts automatically after a set period.
(Contact-form spam has its own playbook; this section is about account creation.)
Thousands of failed logins
Likely cause: credential stuffing, using username and password pairs leaked from other sites.
Fix:
- Tight rate limits on the login URL; Cloudflare gives protecting a login endpoint as a typical use of rate limiting.
- Managed rules: Cloudflare describes its managed rulesets as pre-configured protection against exploits including use of stolen or leaked credentials, and notes a free managed ruleset available on all plans (as of November 2026; other rulesets depend on the plan).
- Two-factor login for staff accounts.
Content scraped and republished
Likely cause: scrapers rotating through IPs and copying whole pages.
Fix:
- Rate limits on product and article URLs.
- A bot-blocking mode or bot score rules, where available.
- Accept that determined scraping of public pages can't be fully stopped; focus on making it expensive.
Strange spikes in "visitors" with no sales
Likely cause: basic bots requesting pages without running JavaScript, or ad-fraud traffic.
Fix:
- Look at your server logs and firewall analytics rather than analytics tags alone.
- Challenge traffic from networks with no real visitors to your business.
Bot "fight" modes: quick wins with trade-offs
Several providers offer one-switch bot protection. Cloudflare's Bot Fight Mode is described as free: it identifies traffic matching known bot patterns and issues computationally expensive challenges to make automated requests costly. The same page lists limits worth knowing before you flip the switch:
- It protects the whole domain and can't be customized or bypassed with WAF custom rules.
- It may challenge API or mobile app traffic.
- It switches on JavaScript detections, which need extra steps if your site uses a Content Security Policy.
If your site has an app or API on the same domain, test carefully, or use rules you can scope to specific paths instead.
Challenges and real people
Every challenge has a cost for humans. Some visitors use screen readers, older devices or privacy tools that fail certain checks. Keep challenges:
- Targeted: on login, sign-up, search and checkout, not every page.
- Non-interactive where possible, so most people never see a puzzle.
- Monitored: watch how many challenges are solved versus failed. A high failure rate from real customers means the rule is too broad.
A starter rule set for a small site
| Problem | Rule | Where |
|---|---|---|
| Fake "Googlebot" | Allow verified bots only | Site-wide setting |
| Login attacks | Rate limit + managed rules | /login, /wp-login.php, account pages |
| Sign-up abuse | Rate limit + challenge | Registration page |
| Search hammering | Rate limit + caching | /search, filtered listings |
| Known exploits | Free managed ruleset | Site-wide |
| Scraping | Rate limit + bot mode | Product or article paths |
Start each new rule in a "log" or "simulate" mode if your provider offers one, watch it for a few days, then switch it to block or challenge.
FAQ
Will blocking bots hurt my SEO?
Not if you allow verified search crawlers. Problems start when rules block by user agent or country without exceptions for real crawlers.
Do I need a paid firewall?
Many small sites manage with free tiers and managed rules from their CDN or host. Check what your current host includes before adding another service.
Can I just block whole countries?
You can, but it's blunt. It may block customers who travel, use VPNs, or live abroad, and it doesn't stop bots routed through other regions.
How do I know a rule is working?
Compare server load, failed logins and sign-up quality before and after, and review the firewall's event log for false positives.
If bots are eating your server's resources, we can help you set up the right rules and check what they catch.
Sources
- https://developers.google.com/search/docs/crawling-indexing/robots/intro
- https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
- https://developers.cloudflare.com/waf/rate-limiting-rules/
- https://developers.cloudflare.com/waf/managed-rules/
- https://developers.cloudflare.com/bots/get-started/bot-fight-mode/
- https://developers.cloudflare.com/bots/concepts/bot/verified-bots/