robots.txt Generator
Create a robots.txt with per-bot rules, AI crawler blocking, crawl-delay and sitemaps.
User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /search Disallow: /*?*sort= Sitemap: https://example.com/sitemap.xml
Place it at the root of your domain (https://example.com/robots.txt). Robots rules are a request, not access control — protect private pages with authentication. Googlebot ignores Crawl-delay.
Test a URL
Blocked for Googlebot — Disallow: /admin/ (group “*”)