Crawl Insights

Crawl Insights gives you visibility into how AI crawlers interact with your website. By analyzing first-party crawl data collected through Cloudflare, Cite AI helps you understand which AI bots access your content, what they crawl, how frequently they visit, and whether they’re able to access your pages successfully.

To get started, navigate to Crawl Insights from the sidebar and connect your Cloudflare account. Once connected, Cite AI automatically identifies supported AI crawlers, categorizes their activity, and continuously updates your dashboard with the latest crawl data.

Dashboard Overview

The Crawl Insights dashboard provides a complete overview of AI crawler activity across your website.

You’ll find:

Filters

Filter crawl activity by:

• Date range
• AI Platform (OpenAI, Google, Anthropic, Perplexity, etc.)
• Bot Type
• Individual Bot

KPI Summary

Quickly monitor your site’s AI crawl health with:

• Bot Visits — Total AI crawler requests.
• Active Bots — Number of unique AI crawlers visiting your website.
• Failure Rate — Percentage of requests returning 4xx or 5xx status codes.

Crawl Activity

Track crawl activity over time to identify trends, spikes, or sudden drops in AI crawler visits.

Switch between:

• Hourly
• Daily
• Weekly
• Monthly

Content Performance

See which folders and URLs receive the most AI crawler activity and identify the content AI platforms visit most frequently.

Crawl Breakdown

Understand your AI traffic from different perspectives.

By Platform

See crawl activity by AI provider, including:

• OpenAI
• Google
• Anthropic
• Perplexity
• Meta
• Other supported platforms

By Bot

View activity for individual crawlers such as:

• GPTBot
• ChatGPT-User
• OAI-SearchBot
• ClaudeBot
• PerplexityBot
• Google-Extended
• Gemini
• And other supported bots.

By Bot Type

AI crawlers are categorized by their purpose:

• Training — Crawlers collecting publicly available content for model training.
• Search — Bots retrieving fresh web content to answer queries.
• User Retrieval — Requests generated while serving a user’s AI search or browsing request.
• Other — Additional supported AI crawler activity.

URL Insights

The URL Insights page provides page-level crawl analytics.

Use filters to analyze crawl activity by:

• Date
• Platform
• Bot
• Bot Type
• Folder
• Status Code
• URL Search

Key Metrics

For the selected filters you’ll see:

• Bot Visits — Total crawler requests.
• Active Bots — Number of unique crawlers.
• Failure Rate — Percentage of unsuccessful crawl requests.
• Top Crawled Folder — Website section receiving the most AI crawler visits.
• Top Crawled URL — Individual page with the highest crawl activity.

Crawled Pages

The URL table provides detailed crawl information for every page visited by AI crawlers.

Each page includes:

Metric	Description
URL	The page visited by the crawler.
Folder	Website section containing the page.
Bot Visits	Total AI crawler requests.
Platforms	AI providers visiting the page.
Status Codes	HTTP response returned to AI crawlers.

Where available, Crawl Insights also connects crawl activity with AI search performance by displaying:

Metric	Description
Retrievals	Number of times the page appeared as a source in tracked AI responses.
Citation Rate	Average citations generated when the page is referenced.
Topics	Number of tracked topics where the page was cited.

These metrics are based on your tracked prompts and are independent of crawler activity.

Settings

Manage your Cloudflare connection from the Settings page.

You can:

• View your connection status.
• Reconnect your Cloudflare account.
• Disconnect Crawl Insights.
• Delete all collected crawl data.

Deleting crawl data is permanent and cannot be undone.

HTTP Status Codes

Crawl Insights records the HTTP response returned to every supported AI crawler.

Common status codes include:

Status	Meaning
200	OK
301	Moved Permanently
302	Temporary Redirect
304	Not Modified
307	Temporary Redirect
308	Permanent Redirect
401	Unauthorized
403	Forbidden
404	Not Found
410	Gone
429	Too Many Requests
500	Internal Server Error
502	Bad Gateway
503	Service Unavailable
504	Gateway Timeout

Monitoring status codes helps identify pages AI crawlers cannot successfully access, allowing you to resolve crawl issues before they impact AI visibility.

Frequently Asked Questions

How do I connect Cloudflare?

Follow the Connect Cloudflare guide in Settings to deploy the Cloudflare Worker and begin collecting AI crawl data.

Can I disconnect Crawl Insights?

Yes. You can disconnect your Cloudflare integration at any time from Settings.

Does my Cloudflare Workers plan affect data collection?

Yes. Cloudflare Workers request limits may affect continuous data collection on high-traffic websites. Upgrading to a paid Workers plan is recommended for uninterrupted tracking.

Can I permanently delete my crawl data?

Yes. Crawl data can be deleted from Settings at any time. This action is irreversible.