And now we have it: ClaudeBot just made 958 requests.

The data

All LLM bots (Mar 17–24, 2026)

ClaudeBot content breakdown

Compare this to last week’s ClaudeBot breakdown: 64% robots.txt/sitemap/topics, 0.8% GEO pages, 0% Q&A pages. The shift from structural monitoring to content ingestion happened in a single week.

ClaudeBot daily breakdown

March 20 is ClaudeBot’s version of the events we’ve now seen from every major crawler. GPTBot had January 7 (547 requests). PerplexityBot had March 10–15 (511 requests across several sessions). ClaudeBot had March 20 (577 requests in one day).

March 20, hour by hour

What we changed (and when)

The ClaudeBot surge didn’t come from nothing. It followed two rounds of infrastructure changes, including one that likely triggered it.

March 9: four content and navigation changes

These are the same changes that preceded PerplexityBot’s activation in article 7.

1. Topic restructuring: 15 topics to 25.

The old topic system had a mega-topic problem. “Genymotion Testing Suite” contained 166 of 178 Q&As (93%). The brand keyword “Genymotion” appeared on 56% of pages, and the clustering algorithm kept grouping everything around it. We filtered brand keywords appearing on more than 25% of pages, increased the topic count from a hardcoded 15 to a dynamic formula based on content volume, and re-ran the pipeline.

Result: the top topic dropped from 166 Q&As (93%) to 87 (49%). Specific user-intent topics emerged: “Apple Silicon Macs” (65 Q&As), “CI/CD Tools” (36 Q&As), “Licensing & Certificates” (17 Q&As). Topics that match how people actually ask questions.

2. Index page revamp.

The old index listed API endpoints and content counts. The new index opens with a product description, then a topic directory with inline summaries for each of the 25 topics. Enough context for a crawler to navigate to relevant content or answer a query directly from the index.

3. Featured Q&A reranking.

The old index.html showed 6 “featured” Q&As in arbitrary database order. The first featured question was “How many free virtual Android devices does Genymotion offer?” We replaced this with retrieval-count-based ranking: the Q&As that ChatGPT-User fetches most often appear first.

PerplexityBot activated one day after these changes. ClaudeBot didn’t. For 11 days, ClaudeBot continued its monitoring loop: robots.txt + sitemap every 1–2 hours, no content pages.

March 20: per-topic sitemaps

On March 20, we deployed a structural change to how the sitemap works.

The old sitemap was a single monolithic sitemap.xml listing ~700 URLs. No semantic structure, no topic grouping, no signal about which content clusters had been updated. ClaudeBot had been reading this sitemap 15 times per week for months without ever committing to a content crawl. The sitemap told it “here are 700 pages” but gave it no way to assess which were fresh or which topics were worth prioritizing.

The new sitemap is a <sitemapindex> pointing to per-topic child sitemaps. Each child sitemap (sitemaps/pricing-and-refunds.xml, sitemaps/android-development-stack.xml, etc.) lists only the pages in that topic, with its own <lastmod> date based on the newest page in the cluster. Topic names in the URLs provide semantic signal: a crawler can see that sitemaps/pricing-billing.xml was updated yesterday without fetching the contents.

We also simplified robots.txt from ~140 lines of redundant per-bot sections down to ~12 lines with a single User-agent: * rule. The llms.txt and llms-full.txt pointers, previously buried at line 90+ after 15 individual bot sections, moved to the top of the file.

The timeline

We can’t prove the per-topic sitemap caused the crawl. But ClaudeBot is the bot that reads structure. It had been checking the old monolithic sitemap 15 times a week for months without crawling. On the day we replaced it with a topic-organized sitemapindex, six hours after deployment, ClaudeBot ran its largest content crawl since December.

Every major crawler has now had its moment

Three bots. Three mass crawl events. Three different structural signals. GPTBot found the site through its sitemap and mass-crawled content. PerplexityBot responded to a redesigned index page with a topic directory. ClaudeBot responded to a sitemapindex organized by topic with per-cluster freshness dates.

ClaudeBot had crawled the site once before, in December (1,267 pages), when it first discovered the domain. Then it went into monitoring mode for three months. The March 20 event brought it back.

The open question: where is Claude-SearchBot?

ClaudeBot made 958 requests. Claude-SearchBot made 0.

This has been true for the entire observation period. Across 60 days of logs, Claude-SearchBot has never visited the AI site. Not once.

For context, OpenAI splits its crawler infrastructure into three bots with distinct roles:

• GPTBot: training data collection
• OAI-SearchBot: builds the retrieval index for ChatGPT’s web search
• ChatGPT-User: real-time retrieval during live conversations

Anthropic has ClaudeBot documented as the training/indexing crawler. Claude-SearchBot is documented as the bot that builds the retrieval index for Claude’s web search feature. We see ClaudeBot. We don’t see Claude-SearchBot.

Three possible explanations:

1. ClaudeBot handles both training and search indexing.

Unlike OpenAI’s split architecture, Anthropic may route all crawl activity through a single bot. If so, the March 20 mass crawl feeds both Claude’s training data and its search retrieval index. We’d have no way to distinguish from log data alone.

2. Claude’s web search uses a partner’s index.

If Claude’s search feature relies on Bing’s index or another third-party search infrastructure rather than a proprietary crawl, then Claude-SearchBot may not need to visit sites directly. ClaudeBot would handle training data only.

3. Claude-SearchBot hasn’t activated for this domain yet.

The training crawl may be a prerequisite. ClaudeBot evaluates the content, and Claude-SearchBot follows later once the domain is flagged as worth indexing for search.

ChatGPT: still steady

ChatGPT-User made 1,154 requests this week. The plateau continues.

Four consecutive weeks at ~1,200. ChatGPT is a reliable citation channel producing roughly 44 content sessions per day, with 40% involving multi-turn follow-up questions. The growth story has moved to other platforms.

PerplexityBot: post-indexing quiet

PerplexityBot dropped from 511 to 58 this week. 45 of those 58 requests were topic pages and robots.txt. Only 2 Q&A pages.

This is normal post-indexing behavior. GPTBot did the same thing after January 7: a mass crawl followed by weeks of 1–2 requests per day. PerplexityBot indexed the content. Now it’s monitoring for changes. The question is whether Perplexity citations start appearing in the next two weeks.

Where we are

Four platforms, four stages:

ChatGPT: steady state. 83% citation rate (last tested March 2). ~1,200 requests/week. The AI site approach is proven here.

Perplexity: indexed, waiting for citations. Deep indexing event completed March 10–15. Citations expected early April.

Claude: just indexed, waiting for everything. First real content crawl March 20. 503 GEO pages, 162 Q&A pages. No Claude-SearchBot traffic. Whether this feeds the citation pipeline or only training data is the open question. Answer expected mid-April.

Gemini: different game entirely. Doesn’t crawl the AI site. Answers from training data and Google Search grounding. Not part of this story.