How to Whitelist Rozzbot in Cloudflare?

Detailed Explanation

We use a web crawler to collect your website data and make it available in Rozz. This article describes steps you can take to ensure that Rozzbot—our crawler—is able to securely index your publicly available content. Due to recent updates from Cloudflare regarding AI bots, including their new feature to block bots that access content for AI model training, we want to guide you on how to allow Rozzbot to continue accessing your website.

Why Whitelist Rozzbot?

Here are a couple of answers to questions you may have about Rozzbot.

Rozzbot:
- Follows your website’s robots.txt file, respecting your preferences on which content can and cannot be accessed.
- Does not scrape unlicensed content for AI training or model development.
- Provides insights after each crawl about some issues we found in your content, such as broken links and duplicates.

With Cloudflare’s recent crackdown on AI bots, it’s important to make sure Rozzbot isn’t blocked on your site. Here’s how to do it.

How to Whitelist Rozzbot in Cloudflare

Create a custom rule in Cloudflare’s Web Application Firewall (WAF) using the steps below:

1. In your Cloudflare dashboard, click on Security in the main menu.
2. Under Security, click on WAF to access your firewall settings.
3. Click Firewall Rules and Select Create a Firewall Rule.
4. Enter a name for the rule, such as “Allow Rozzbot.”
5. Set Field to User Agent from the dropdown.
6. Set Operator to Equals.
7. Set Value to ‘Rozzbot’.
8. Set ‘Then…’ to ‘Allow’.
9. Click Deploy.

That’s It!