Claude User Agents

Claude's primary user agent string is 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude/1.0; https://claude.ai/)'. This identifier is used when Claude browses web content through its web browsing capability.

Introduction to Claude User Agents

Anthropic's Claude AI assistant uses specific user agents when crawling and accessing web content. Understanding these user agents is crucial for website owners who want to optimize their content for Claude or control how their content is accessed by Anthropic's systems.

Anthropic documents three primary Claude-related bots in their official help center:

ClaudeBot – training crawler that collects web content for model development
Claude-User – agent used when individual Claude users fetch web pages
Claude-SearchBot – crawler that indexes content to improve search result quality

See Anthropic's guidance in their official article on Claude bots and robots.txt controls.

What is Claude's Main User Agent String?

Claude typically identifies itself with user agents that contain "Claude" or "Anthropic" in the string. Here are the known Claude user agents:

Main Claude User Agent

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude/1.0; https://claude.ai/)

This is the primary user agent used by Claude when browsing the web through its web browsing capability.

Key Claude Bots and Their Roles

In practice, you'll most commonly see the following Claude-related bots in your proxy and server logs:

ClaudeBot – Anthropic's primary training crawler. When this bot is allowed, your content may be used to improve future Claude models.
Claude-User – represents real people using Claude who trigger page fetches while asking questions. This is the best signal that Claude is actively recommending your content to users.
Claude-SearchBot – focuses on indexing and evaluating content quality to improve Claude's search-style answers.
claude-web – older identifier that some sites still see in logs; xSeek continues to detect and classify it for backwards compatibility.

What are the Different Ways Claude Accesses Web Content?

Claude can access web content in several ways:

Direct user requests - When a user asks Claude to retrieve information from a specific URL
Background information gathering - When Claude needs to supplement its knowledge
Training data collection - For improving Claude's capabilities (subject to robots.txt directives)

How Can I Control Claude's Access to My Content?

Website owners can control how Claude accesses their content through:

Robots.txt Configuration

You can use the following directives in your robots.txt file to control each Claude bot separately, following Anthropic's recommendations:

# Block Claude training crawler (ClaudeBot) from your entire site
User-agent: ClaudeBot
Disallow: /

# Allow user-initiated Claude access but block training
User-agent: Claude-User
Disallow:

User-agent: ClaudeBot
Disallow: /

# Block Claude-SearchBot from specific sections only
User-agent: Claude-SearchBot
Disallow: /private/
Disallow: /members-only/

# Optionally slow down crawling
User-agent: ClaudeBot
Crawl-delay: 1

How Can I Optimize Content for Claude?

To ensure your content performs well when accessed by Claude:

Use clear, well-structured HTML with proper semantic markup
Ensure content is accessible and doesn't rely solely on JavaScript for rendering
Provide clear context and comprehensive information on topics
Include metadata that helps Claude understand your content's purpose and authority

How Can I Track Claude Visits?

With xseek, you can track when and how Claude accesses your content:

Monitor Claude user agent visits in your analytics dashboard
Track which content Claude is accessing most frequently
Analyze how Claude interprets and represents your content in responses

FAQ

What is Claude's main user agent string?

What are the different ways Claude accesses web content?

Claude accesses web content through direct user requests when asked to retrieve information from specific URLs, and through background information gathering. These methods allow Claude to browse and analyze web content as needed.

How can I detect if Claude is accessing my website?

You can identify Claude's access by looking for 'Claude' or 'Anthropic' in the user agent string. The main identifier contains 'Claude/1.0' and 'claude.ai' in the string, making it easily distinguishable from other web crawlers.

Related User Agents

Learn about other AI user agents to better manage your website's interaction with AI systems:

OpenAI User Agents - ChatGPT, GPTBot, and OAI-SearchBot
Perplexity User Agents - Perplexity AI search engine
Deepseek User Agents - Deepseek AI
Llama User Agents - Meta's Llama AI
Bing AI User Agents - Bing AI