Table of Contents
I once accidentally told every search engine on Earth to ignore my entire website. One wrong line in a file called robots.txt, and my site disappeared from Google. I caught it fast, but if I hadn’t been monitoring, it could have stayed that way for weeks. These kinds of mistakes took minutes to make and could take months to recover from protecting your site with backups.
Robots.txt is the most powerful and dangerous file most website owners have never heard of. It controls whether search engines can see your content. Configured correctly, it protects private areas while letting Google index everything that matters. Configured incorrectly, it can erase your entire online presence overnight the technology of writing hub.
What Robots.txt Does
When search engines visit your website, they check a file at yoursite.com/robots.txt before crawling anything else. This file contains instructions about which parts of your site they’re allowed to access. For more, see your web designer wrote your website copy and it shows. It follows the Robots Exclusion Protocol, which every major search engine respects. For more, see wordPress reality.
A properly configured robots.txt file directs search engines to your important content while keeping them away from admin areas, private directories, and pages that create duplicate content problems. It can improve your SEO by focusing crawl resources on the pages that matter. It can also destroy your search visibility if you get it wrong.
The critical thing to understand: robots.txt is a polite request, not a security measure. Legitimate search engines follow these rules. Malicious bots, scrapers, and hackers ignore them completely. If you need actual security for sensitive content, use password protection or server-level blocks. Robots.txt is for managing search engine behavior, not for protecting private data.
How Mistakes Happen
Every character in a robots.txt file matters. The difference between blocking your admin folder and blocking your entire website is a single word:
Disallow: /wp-admin/ blocks your admin folder. Correct.
Disallow: / blocks your entire website. Catastrophic.
Other common mistakes: blocking /blog/ when you meant to block /blog/drafts/. Forgetting that directives are case sensitive. Missing the space between Disallow: and the path. Deploying a development robots.txt file to your live site. Each of these can remove pages or entire sections from search results, and recovery takes far longer than the mistake took to make.
Safe Robots.txt Examples
Basic WordPress Site
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xmlThis blocks WordPress admin areas that shouldn’t be indexed while allowing a necessary AJAX file. Conservative and safe. Start here.
Small Business Site
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /temp/
Sitemap: https://yourbusiness.com/sitemap.xmlProtects admin areas and private content while leaving all public pages accessible.
E-commerce Site
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Sitemap: https://yourstore.com/sitemap.xmlBlocks transactional pages that create duplicate content issues while protecting customer data.
Development Site (Temporary)
User-agent: *
Disallow: /
Sitemap: https://devsite.com/sitemap.xmlBlocks everything. Use only on development sites. Remove immediately when going live. This is the line that will destroy your search visibility if it accidentally ends up on your production site.
Advanced Techniques
Targeting Specific Search Engines
User-agent: Googlebot
Disallow: /premium-content/
User-agent: AhrefsBot
Disallow: /
User-agent: *
Disallow: /admin/Different rules for different crawlers. This example blocks aggressive SEO tool crawlers entirely while allowing Google access to most content.
Pattern Blocking with Wildcards
User-agent: *
Disallow: /*?print=
Disallow: /*&utm_source=
Disallow: /category/*/page/Blocks URL patterns that create duplicate content or waste crawl budget without blocking individual pages.
Controlling AI Crawlers
Robots.txt is evolving to address AI-powered bots that scrape content for training language models and generating AI answers. This is a rapidly changing area, but there are already practical directives you can use.
Google and Apple have introduced specific user agents for their AI training crawlers, separate from their search crawlers. This means you can block AI training without affecting your search visibility:
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /These directives tell Google and Apple not to use your content for AI model training while still allowing their regular search crawlers to index your site normally.
Cloudflare has expanded robots.txt further with Content-Signal directives that provide granular control over how AI bots use your content. These include search=yes/no to control whether content can build search indexes, ai-input=yes/no to determine if content can be used as input for AI-generated answers like chatbot responses, and ai-train=yes/no to block or allow content for training AI models. The defaults are search=yes, ai-train=no, and ai-input=neutral until the site owner sets them.
Emerging standards like llms.txt and tools like Spawning’s Do Not Train protocol offer additional layers of control at the content level rather than the site level.
The critical caveat: none of robots.txt is enforced by all bots. This applies to every directive in the file, not just the AI-specific ones. AI companies may ignore them entirely. These directives function primarily as legal reservations of rights and best-effort signals. They should be combined with other protections like firewalls and access controls if you’re serious about preventing AI scraping.
Testing Before You Deploy
Never upload a robots.txt file without testing it first. Use Google Search Console’s robots.txt tester to check syntax and verify that your rules do what you intend. Test multiple URLs, including your homepage, your most important content pages, and the pages you’re trying to block. Verify that your sitemap location is correct.
Back up your current robots.txt file before making changes. Start with minimal restrictions and add rules gradually. Upload during low-traffic hours. Monitor crawl errors and traffic data for at least 48 hours after any change. If something looks wrong, revert to your backup immediately.
When Something Goes Wrong
If you discover a robots.txt error that’s blocking content from search engines, fix the file immediately. Then request a re-crawl in Google Search Console and resubmit your sitemap. Monitor crawl errors daily for the first week. Check the indexing status of pages that were blocked. Submit individual URLs for re-indexing if necessary.
Recovery time depends on how long the error was live and how much content was affected. A few hours of blocking might resolve within days. Weeks of blocking can take months to fully recover from. For more, see protecting your site with backups. The longer bad directives stay in place, the more damage they cause and the longer the recovery takes.
WordPress Specifics
WordPress automatically generates a basic robots.txt file with minimal protection for admin areas. This default is safe but basic. Most serious WordPress sites benefit from a customized file. My WordPress help for writers exists for exactly this. SEO plugins like SEOpress let you edit robots.txt directly from your admin dashboard, which makes changes easier but also easier to break.
Don’t block /wp-content/uploads/ where your images are stored. Blocking this directory kills your image SEO. Do block /wp-admin/ and /wp-includes/. Always allow /wp-admin/admin-ajax.php which WordPress needs for core functionality.
The Point
Most successful websites use conservative robots.txt files that protect obvious admin areas and leave everything else accessible. The goal isn’t sophistication. It’s a file that safely serves your business without risking the kind of mistake that removes your site from search results.
Start simple. Test everything. Monitor after changes. When in doubt, allow access rather than risk blocking important content. The most dangerous robots.txt file is the one someone edited without testing.