Free Robots.txt Generator Tool.

A robots.txt file is a plain-text file placed at the root of your website—for example, https://yourwebsite.com/robots.txt — that instructs web crawlers and bots which pages or directories they are permitted or prohibited from accessing. It forms part of the Robots Exclusion Protocol, a widely accepted standard observed by all major search engines, including Google, Bing, and Yahoo.

For bloggers, publishers, and website owners, a correctly configured robots.txt file is one of the most cost-effective SEO improvements available. Without it, search engine crawlers may waste their crawl budget on low-value pages such as admin panels, tag archives, or internal search results—time and resources that could instead be spent indexing your best content.

Quick Fact: Google has confirmed that crawl budget matters for large websites. Even for smaller sites, unnecessary crawler activity on duplicate or private pages can dilute how efficiently your content is indexed.

How to Use This Free Robots.txt Generator

Understanding the Robots.txt Syntax

The structure of a robots.txt file is straightforward. Each instruction block begins with a User-agent directive identifying the crawler, followed by one or more Allow or disallow directives specifying which paths the crawler may or may not access. A blank line separates each block.

# Allow all crawlers full access, except specific directories
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
Allow: /

# Block OpenAI's GPTBot from training on your content
User-agent: GPTBot
Disallow: /

# Block Google's AI training crawler
User-agent: Google-Extended
Disallow: /

# Sitemap location
Sitemap: https://www.yourwebsite.com/sitemap.xml

The * wildcard in User-agent: * applies the rules to every crawler that has not been given specific instructions elsewhere in the file. Rules listed under a specific user-agent name take precedence over wildcard rules for that particular bot.

Blocking AI and LLM Crawlers: Should You Opt Out?

Since 2023, a growing number of AI companies have deployed crawlers to scrape publicly accessible web content for training large language models. This has prompted considerable debate among publishers and content creators about whether — and how — to restrict such access.

Unlike search engine crawlers, AI training bots typically offer no direct benefit to your site’s discoverability or traffic. If your content is scraped for model training, you receive no attribution, no backlink, and no visitor. For this reason, many publishers are choosing to opt out by blocking known AI crawlers via robots.txt file.

Important: Blocking GPTBot does not block OAI-SearchBot, which powers ChatGPT’s search results. If you wish to exclude your site from both training data and ChatGPT search results, you must list each user-agent separately. Our generator handles this automatically when you select multiple bots.

Common Robots.txt Mistakes to Avoid

1. Blocking Your Entire Site by Accident: The most damaging error is accidentally disallowing all crawlers from all content. The directive Disallow: / under User-agent: * will prevent every search engine from indexing your site. Always verify that your global wildcard block includes an explicit Allow: / after any specific disallow rules.
2. Using Robots.txt to Hide Sensitive Data: Robots.txt is a publicly accessible file. Any path you list in a Disallow directive is, by definition, visible to anyone who reads your robots.txt. For genuinely sensitive content, use server-side authentication, not robots.txt directives.
3. Blocking CSS and JavaScript Files: Googlebot renders pages visually and needs access to your CSS and JavaScript in order to accurately assess your page content and layout. Blocking these file types can negatively affect how your pages are understood and ranked.
4. Placing the File in the Wrong Location: Your robots.txt file must reside at the root of your domain: https://yourdomain.com/robots.txt. A file placed in a subdirectory such as this /blog/robots.txt will not be recognized by crawlers.
5. Assuming All Bots Comply: Reputable crawlers from Google, Bing, and most legitimate SEO tools respect robots.txt. However, malicious bots and scrapers typically do not. Robots.txt is a directive, not a technical barrier — use additional security measures for true access control.

How to Add Robots.txt to a WordPress Site

WordPress 5.3 and later generates a virtual robots.txt file automatically via its built-in virtual file handling. There are two primary approaches for customizing it.

• Option A — Via Yoast SEO or Rank Math: Both plugins provide a built-in robots.txt editor under their respective settings menus. Paste the output from our generator directly into the editor and save. This is the recommended method, as it avoids directly editing server files.
• Option B — Upload directly to your server: Download the robots.txt file produced by our generator and upload it to the root directory of your hosting account using FTP or your hosting control panel’s file manager. Ensure it is placed alongside your wp-config.php file, not inside the /wp-content/ folder.

Tip: After uploading or saving your robots.txt, verify it is accessible by visiting https://yourdomain.com/robots.txt in your browser. You can also test it using Google Search Console’s Robots.txt Tester tool.

Frequently Asked Questions on Robots.txt Generator