Ruthenium IT

How to Stop AI Bots from Scraping Your Website Content Using Cloudflare

In the age of artificial intelligence, content scraping has taken on a whole new dimension. AI companies and rogue actors deploy bots to crawl websites and harvest valuable content—often without permission. If you’re a content creator, blogger, or business running a WordPress site, this could mean your original work is being used to train large language models without your consent.

The good news? If you use Cloudflare, you have powerful tools at your disposal to stop this.

🚨 Why You Should Block AI Scrapers

• Protect your intellectual property: Original blog posts, articles, and creative content can be used to train AI systems.
• Avoid server strain: AI bots can flood your site with unnecessary requests.
• Preserve your SEO rankings: If your content appears elsewhere, Google may penalize your rankings.
• Control how your data is used: You should have a say in whether your content is ingested into AI databases.

---

🛡️ Step-by-Step: Blocking AI Bots with Cloudflare

✅ Where to Find AI Bot Blocking Now

1. Block AI Bots Setting (New Dashboard)

1. Log in to your Cloudflare account and select your domain.
2. Navigate to Security → Settings.
3. Use the search/filter field and type “Bot traffic”.
4. You should see a “Block AI Bots” configuration option.
   • Click the edit icon, then choose Block on all pages, Block only on ad-hosting hostnames, or Do not block

This replaces the previous straightforward toggle that appeared under “Bots” in the old dashboard.

---

Bot Fight Mode / Super Bot Fight Mode

These settings are still valid and help protect against general automation:

• On Free accounts: Bot Fight Mode can be enabled via Security → Bots in the old interface—or via Settings filter in the new one
• Paid plans (Pro/Business) can enable Super Bot Fight Mode for enhanced detection and JavaScript challenges

If you’re not seeing these at all, it may be that you’re already using the updated Security → Settings → Bot traffic path.

---

2. Add Firewall Rules to Block Specific AI Crawlers

Many AI-related crawlers identify themselves via their user agents. You can block these manually.

To create a firewall rule:

1. Go to Security > WAF > Custom Rules in Cloudflare.
2. Click Create a Firewall Rule.
3. Name it something like Block AI Scrapers.
4. Set the expression to block known user agents:

plaintextCopyEdit(http.user_agent contains "Bytespider") or
(http.user_agent contains "ChatGPT-User") or
(http.user_agent contains "OpenAI") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "Anthropic") or
(http.user_agent contains "Amazonbot") or
(http.user_agent contains "GPTBot") or
(http.user_agent contains "AIEngine")

5. Action: Block
6. Click Deploy.

You can update this list as new AI-related bots appear. Check their documentation or server logs for more identifiers.

---

3. Use Cloudflare’s Verified Bot List (Optional)

If you want to allow only known and legitimate bots (like Googlebot or Bingbot):

• Go to Security > WAF.
• Use Bot Management to filter traffic based on verified bot status.
• Create a rule that challenges or blocks non-verified bots.

This allows Google to crawl your site while keeping AI scrapers out.

---

4. Block Known IP Ranges (Advanced)

Some scrapers don’t identify themselves and use cloud servers (AWS, Azure, etc.) to crawl your site.

• Use Cloudflare Analytics > Traffic > IPs to monitor suspicious IPs.
• Add persistent IP blocks for malicious traffic under Tools > IP Access Rules.

Note: IPs can change frequently. Use with care.

---

✋ Bonus: Use a robots.txt File to Disallow Scraping (Not Foolproof)

You can signal to bots that they shouldn’t scrape your site:

plaintextCopyEditUser-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /private/

However, rogue bots can and do ignore robots.txt.

---

✅ Conclusion

Protecting your WordPress content from AI bots is essential in today’s digital environment. While you can’t stop every scraper, Cloudflare gives you powerful tools to reduce the risk dramatically. By combining Bot Fight Mode, custom firewall rules, and good monitoring habits, you can keep your content safe, reduce server strain, and retain control over how your work is used.

---

Want help securing your WordPress site further?
Drop a comment or get in touch—we’re here to help.