findXpand
All toolsRobots, Sitemap & Crawling

Robots.txt Generator

Build a robots.txt with per-bot rules and one-click toggles for AI crawlers.

Generate a robots.txt file, choose which paths to disallow, and toggle access for search and AI crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) individually.

AI crawlers — toggle which ones may access the site
robots.txt
User-agent: *
Disallow: /admin
Disallow: /cart

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: OAI-SearchBot
Allow: /

Rather not paste this in by hand? findXpand checks every page of your site for SEO, AI search, speed, accessibility and security, then writes the fixes and puts them live for you.

Scan your site