Generate a customized robots.txt file to control which AI crawlers and search engines can access your website. Essential for AI SEO and Generative Engine Optimization (GEO).
Set the default behavior for all crawlers
Specify paths that should be blocked for all crawlers
Trains ChatGPT models
Powers ChatGPT web search
Fetches shared links
Claude AI crawler
Claude training data
Fresh web content
AI search index
Gemini AI
Alexa & recommendations
Apple AI training
TikTok's AI
Private AI answers
Enterprise LLMs
Meta AI crawler
French AI company
Google Search
Google Images
Google Mobile Search
Microsoft Bing
Preview and download your robots.txt file
Test your robots.txt in Google Search Console before deploying
Monitor AI crawler activity to track which bots visit your site
Update your robots.txt quarterly as new AI crawlers emerge
Balance access and protection for optimal visibility