Queldrex
All Tools
Free Tool · AI Visibility

robots.txt Generator

Generate a robots.txt file with precise control over which AI crawlers can access your site. By default, all major AI crawlers are allowed — which is what you want for maximum AI visibility.

General Access

Disallowed Paths

One path per line. These paths are blocked from all crawlers.

AI Crawlers

·

GPTBot

OpenAI / ChatGPT

ChatGPT-User

ChatGPT browsing plugin

ClaudeBot

Anthropic / Claude

Claude-Web

Claude web search

PerplexityBot

Perplexity AI

Google-Extended

Google Gemini / AI Overviews

Googlebot

Google Search

Bingbot

Microsoft Bing / Copilot

CCBot

Common Crawl (training data)

cohere-ai

Cohere AI

anthropic-ai

Anthropic AI training

Omgilibot

Meta / Llama training

Sitemap URL

robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/

How to deploy

Save this as robots.txt and upload it to the root of your website so it's accessible at yoursite.com/robots.txt.

Check your current robots.txt

The AI Visibility Scanner audits your robots.txt as one of 14 signals.

Free scan →

How It Works

Control exactly which AI crawlers and search bots can access your site — with point-and-click simplicity.

01
Toggle crawlers
Check or uncheck each AI crawler (GPTBot, ClaudeBot, Bingbot, etc.) and search crawler. Allow or block each one independently.
02
Add a sitemap
Enter your sitemap URL (e.g. yoursite.com/sitemap.xml). This tells all crawlers where to find your content efficiently.
03
Download & upload
Copy or download the generated robots.txt file and upload it to your site root at yoursite.com/robots.txt.

Example output — block AI training bots, allow search engines

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Allow search engine indexing
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Sitemap: https://yoursite.com/sitemap.xml