Free Online Robots.txt Generator

A robots.txt file is a plain text file placed in the root directory of your website (e.g., example.com/robots.txt) that tells search engine crawlers which pages and directories they can or cannot access. It follows the Robots Exclusion Protocol, a standard that all major search engines respect. The file contains rules organized by user-agent (the name of the crawler) with Allow and Disallow directives that control access to specific URL paths.

Use this generator to build your rules visually. Select which crawlers you want to target, add Allow or Disallow directives for each path, include your sitemap URL, and copy or download the generated file. Upload the file to the root directory of your website so it is accessible at yourdomain.com/robots.txt. Most web hosts let you upload files via FTP, cPanel File Manager, or your CMS settings.

The robots.txt file must be placed in the root directory of your website and served at the exact URL yourdomain.com/robots.txt. Search engines only look for it at that specific location. If your site uses subdomains, each subdomain needs its own robots.txt file. On most hosting platforms, the root directory is called public_html, www, or public.

The Disallow directive tells a crawler not to access a specific URL path. For example, Disallow: /admin/ prevents the specified bot from crawling anything under the /admin/ directory. An empty Disallow value (Disallow: with nothing after it) means allow everything, which is a common source of confusion. To block an entire site, use Disallow: / which blocks all paths starting from the root.

Yes. Adding a Sitemap directive (e.g., Sitemap: https://example.com/sitemap.xml) helps search engines discover your XML sitemap even if it has not been submitted through Google Search Console or Bing Webmaster Tools. The Sitemap directive is placed outside any user-agent group because it applies globally to all crawlers. You can list multiple sitemaps if your site has more than one.

Robots.txt affects SEO indirectly by controlling crawl budget and page discovery. It does not prevent pages from being indexed if other sites link to them. To prevent indexing, use a meta robots noindex tag instead. However, a misconfigured robots.txt that blocks Googlebot from important pages will cause those pages to disappear from search results, which can devastate organic traffic.

Robots.txt controls crawling, which is whether a bot visits your page at all. Meta robots tags control indexing, which is whether a page appears in search results after being crawled. Robots.txt blocks access at the server level before the crawler downloads the page. A noindex meta tag requires the crawler to visit the page first and then instructs it not to list the page in results. Use robots.txt for directories you never want crawled and meta robots for pages that can be crawled but should not appear in search results.

Add separate user-agent blocks for each AI crawler you want to block. Common AI bot names include GPTBot (OpenAI), ChatGPT-User, Google-Extended (Gemini training data), CCBot (Common Crawl), and anthropic-ai (Claude). For each one, add User-agent: followed by the bot name and Disallow: / on the next line. This generator includes a Block AI Bots preset that sets up all these rules in one click.

Yes, you can block any path including image and CSS directories. However, Google strongly recommends against blocking CSS, JavaScript, and image files because Googlebot needs to render your pages to understand their content and layout. Blocking these resources can hurt your search rankings because Google cannot properly evaluate the user experience of your pages.

Use Google Search Console's robots.txt Tester tool to check whether specific URLs are blocked or allowed by your rules. Enter the URL you want to test and the tool shows which rule matches. You can also check your live robots.txt by visiting yourdomain.com/robots.txt in a browser. After deploying changes, allow a few days for search engines to re-fetch the file.