Connect Cloudflare

Connect your Cloudflare account to start tracking AI crawler activity across your website. Once connected, Cite AI captures requests from supported AI bots in real time, helping you understand how AI search engines discover, crawl, and interact with your content.

Cite AI only stores requests from supported AI crawlers. All other website traffic is ignored.

Connecting Cloudflare

Cite AI uses a Cloudflare Worker to capture AI crawler requests and securely forward them for analysis without affecting your website’s performance.

Before You Begin

You’ll need:

• A Cloudflare account with access to your website.
• Permission to create API tokens and deploy Workers.

Step 1: Create a Cloudflare API Token

1. Log in to your Cloudflare dashboard.
2. Open My Profile → API Tokens.
3. Click Create Token.
4. Select Create Custom Token.
5. Give your token a descriptive name.

Add the following permissions:

Resource	Permission
Workers Scripts	Edit
Zone	Read
Workers Routes	Edit

After adding the permissions, review your settings and click Create Token.

Step 2: Connect Your Account

1. In Cite AI, open Settings → Crawl Insights → Cloudflare.
2. Paste your Cloudflare API token.
3. Click Validate Token.
4. Select the Cloudflare Zone you want to monitor.
5. Click Deploy Worker.

Once deployment completes, the Worker immediately begins monitoring supported AI crawler traffic.

What Happens Next?

After the Worker is deployed, Cite AI automatically:

• Detects supported AI crawlers visiting your website.
• Captures crawl activity in real time.
• Filters out all non-AI traffic.
• Populates your Crawl Insights dashboard with crawl activity, trends, and page-level insights.

Historical data begins building as new crawler requests are received.

Cloudflare Worker Limits

The Cloudflare Worker processes every request made to your website, including requests from AI crawlers.

Cloudflare’s free Workers plan includes a daily request limit. If this limit is exceeded:

• Crawl data collection pauses until the quota resets.
• Your website may return Cloudflare errors if the Worker is configured to Fail Closed.

To avoid interruptions, we recommend:

• Setting the Worker route to Fail Open so your website remains accessible if limits are reached.
• Upgrading to a Cloudflare Workers Paid plan for uninterrupted data collection on higher-traffic websites.

Supported AI Crawlers

Cite AI automatically detects requests from major AI crawlers and agents, including:

• GPTBot
• ChatGPT-User
• OAI-SearchBot
• ClaudeBot
• Claude-SearchBot
• Gemini
• Google-Extended
• PerplexityBot
• GrokBot
• Applebot
• Amazonbot
• DeepSeekBot
• Mistral AI
• Meta AI
• Cohere
• And other supported AI crawlers.

Only supported AI crawler requests are stored and analyzed. Regular website traffic is never collected.
:::