Free Online Robots.txt Generator
Generate and validate robots.txt files for search engine crawlers
Quick Start Presets
Full URL to your XML sitemap. Helps search engines discover your pages.
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
robots.txt controls how search engines crawl your site. Incorrect rules can prevent pages from appearing in search results. Test your file using Google Search Console before deploying to production.
Try these next
Why use Robots.txt Generator
- Web developers setting up new projects generate a correct robots.txt in seconds instead of looking up syntax documentation every time.
- SEO consultants auditing client sites quickly spot misconfigured crawl rules and generate corrected files with proper Allow and Disallow directives.
- Site owners concerned about AI training data use the Block AI Bots preset to block GPTBot, CCBot, and other LLM crawlers in one click while keeping search engines active.
- WordPress and Shopify users who lack server administration experience build properly formatted rules without memorizing the robots.txt specification.
- Marketing teams deploying staging sites use the Block All preset to prevent search engines from indexing test environments before launch.
How it works
The generator converts your form selections into valid robots.txt syntax following the Robots Exclusion Protocol. Each rule group maps to a User-agent line followed by Allow and Disallow directives for the paths you specify. Directives are written in the order you add them, with the most specific path winning when multiple rules match the same URL. An optional Crawl-delay value is appended per group for crawlers that support request throttling. The Sitemap directive is placed at the end of the file outside any group because it applies globally. Built-in validation checks for dangerous patterns like Disallow: / under User-agent: *, duplicate user-agent groups that could confuse crawlers, and sitemap URLs missing the https:// protocol. The output updates reactively as you edit, so you always see the exact file content before copying or downloading.
About this tool
Generate a correctly formatted robots.txt file for your website with this free online generator. A robots.txt file sits in the root directory of your domain and tells search engine crawlers which pages and directories they are allowed or not allowed to access. Every website that cares about organic search performance needs a properly configured robots.txt file to manage crawl budget, keep private directories out of search indexes, and direct bots toward valuable content. Getting the syntax wrong can have serious consequences, from accidentally blocking Googlebot from your entire site to leaving staging environments exposed in search results. This generator lets you build rules visually without memorizing robots.txt syntax. Select user-agents from a dropdown that includes Googlebot, Bingbot, Yandex, and modern AI crawlers like GPTBot, ChatGPT-User, Google-Extended, CCBot, and anthropic-ai. Add Allow and Disallow directives for each bot, set an optional crawl-delay for crawlers that support it, and include your sitemap URL so search engines can discover all your pages. Quick-start presets let you generate common configurations in one click: Allow All for a standard open site, Block All for staging environments, or Block AI Bots to prevent large language models from training on your content while keeping search engines active. The output updates in real time as you type, and built-in warnings alert you to dangerous patterns like blocking all crawlers from your root path. Copy the finished file with one click or download it as robots.txt ready to deploy. All processing happens in your browser with no server interaction, no sign-up, and no data stored.
How to use Robots.txt Generator
- Choose a preset or start from scratch. Select a quick-start preset like Allow All, Block All, or Block AI Bots to pre-fill common configurations, or start with the default settings and customize from there.
- Add user-agent rules. Pick a crawler from the dropdown (Googlebot, Bingbot, GPTBot, or any custom bot name) and add Allow or Disallow directives for the paths you want to control.
- Add your sitemap URL. Enter the full URL of your XML sitemap so search engines can discover all your pages. This is added as a global Sitemap directive at the end of the file.
- Review warnings and output. Check the live preview for your generated robots.txt content. The tool warns you about dangerous patterns like blocking all crawlers from your entire site.
- Copy or download. Click Copy to grab the content for pasting, or Download to save it as a robots.txt file. Upload it to the root directory of your website.
Use cases
- A startup founder launching a new SaaS product generates a robots.txt that allows search engines to crawl all public pages while blocking the /api/ and /admin/ paths from being indexed.
- An SEO agency managing dozens of client sites uses the tool to quickly produce standardized robots.txt files during site audits rather than writing them from memory.
- A content publisher who does not want AI models trained on their articles uses the Block AI Bots preset to block GPTBot, ChatGPT-User, and CCBot without affecting Google or Bing rankings.
- A developer preparing a staging environment uses the Block All preset to generate a robots.txt that keeps the entire test site out of search results until launch day.
- A small business owner migrating from one CMS to another creates a new robots.txt that matches the directory structure of the new platform and includes the updated sitemap URL.
Frequently Asked Questions
A robots.txt file is a plain text file placed in the root directory of your website (e.g., example.com/robots.txt) that tells search engine crawlers which pages and directories they can or cannot access. It follows the Robots Exclusion Protocol, a standard that all major search engines respect. The file contains rules organized by user-agent (the name of the crawler) with Allow and Disallow directives that control access to specific URL paths.
Use this generator to build your rules visually. Select which crawlers you want to target, add Allow or Disallow directives for each path, include your sitemap URL, and copy or download the generated file. Upload the file to the root directory of your website so it is accessible at yourdomain.com/robots.txt. Most web hosts let you upload files via FTP, cPanel File Manager, or your CMS settings.
The robots.txt file must be placed in the root directory of your website and served at the exact URL yourdomain.com/robots.txt. Search engines only look for it at that specific location. If your site uses subdomains, each subdomain needs its own robots.txt file. On most hosting platforms, the root directory is called public_html, www, or public.
The Disallow directive tells a crawler not to access a specific URL path. For example, Disallow: /admin/ prevents the specified bot from crawling anything under the /admin/ directory. An empty Disallow value (Disallow: with nothing after it) means allow everything, which is a common source of confusion. To block an entire site, use Disallow: / which blocks all paths starting from the root.
Yes. Adding a Sitemap directive (e.g., Sitemap: https://example.com/sitemap.xml) helps search engines discover your XML sitemap even if it has not been submitted through Google Search Console or Bing Webmaster Tools. The Sitemap directive is placed outside any user-agent group because it applies globally to all crawlers. You can list multiple sitemaps if your site has more than one.
Robots.txt affects SEO indirectly by controlling crawl budget and page discovery. It does not prevent pages from being indexed if other sites link to them. To prevent indexing, use a meta robots noindex tag instead. However, a misconfigured robots.txt that blocks Googlebot from important pages will cause those pages to disappear from search results, which can devastate organic traffic.
Robots.txt controls crawling, which is whether a bot visits your page at all. Meta robots tags control indexing, which is whether a page appears in search results after being crawled. Robots.txt blocks access at the server level before the crawler downloads the page. A noindex meta tag requires the crawler to visit the page first and then instructs it not to list the page in results. Use robots.txt for directories you never want crawled and meta robots for pages that can be crawled but should not appear in search results.
Add separate user-agent blocks for each AI crawler you want to block. Common AI bot names include GPTBot (OpenAI), ChatGPT-User, Google-Extended (Gemini training data), CCBot (Common Crawl), and anthropic-ai (Claude). For each one, add User-agent: followed by the bot name and Disallow: / on the next line. This generator includes a Block AI Bots preset that sets up all these rules in one click.
Yes, you can block any path including image and CSS directories. However, Google strongly recommends against blocking CSS, JavaScript, and image files because Googlebot needs to render your pages to understand their content and layout. Blocking these resources can hurt your search rankings because Google cannot properly evaluate the user experience of your pages.
Use Google Search Console's robots.txt Tester tool to check whether specific URLs are blocked or allowed by your rules. Enter the URL you want to test and the tool shows which rule matches. You can also check your live robots.txt by visiting yourdomain.com/robots.txt in a browser. After deploying changes, allow a few days for search engines to re-fetch the file.
Related Tools
Discover more free utilities to enhance your productivity.
Meta Tag Generator
Generate SEO meta tags and Open Graph tags
.htaccess Redirect Generator
Generate 301/302 redirect rules for Apache .htaccess files
Text to Slug Converter
Convert any text into a clean URL-friendly slug
Cron Expression Generator
Build and decode cron schedule expressions