I once accidentally told every search engine on Earth to ignore my entire website. One wrong line in a file called robots.txt, and my site disappeared from Google.
I caught it fast, but if I hadn’t been monitoring, it could have stayed that way for weeks. These kinds of mistakes took minutes to make and could take months to recover from.
Robots.txt is the most powerful and dangerous file most website owners have never heard of. It controls whether search engines can see your content. Configured correctly, it protects private areas while letting Google index everything that matters. Configured incorrectly, it can erase your entire online presence overnight. There’s more on this in the technology of writing hub.
I think robots.txt deserves a lot more fear than it gets. Site owners obsess over fonts and page builders and never look at the one file that can switch off their search traffic, and too many SEO plugins and hosting dashboards let you change it without a word of warning about what you just did.
What Does Robots.txt Do?
When search engines visit your website, they check a file at yoursite.com/robots.txt before crawling anything else. This file contains instructions about which parts of your site they’re allowed to access. It follows the Robots Exclusion Protocol, which every major search engine respects. For more, see your web designer wrote your website copy and it shows and wordPress reality.
Configured right it protects private areas; configured wrong it erases your online presence overnight.Share on X
A properly configured robots.txt file directs search engines to your important content while keeping them away from admin areas, private directories, and pages that create duplicate content problems. It can improve your SEO by focusing crawl resources on the pages that matter. Get it wrong and it can destroy your search visibility.
Robots.txt is the most powerful and dangerous file most website owners have never heard of.
The critical thing to understand: robots.txt is a polite request with no enforcement behind it. Legitimate search engines follow these rules. Malicious bots, scrapers, and hackers ignore them completely. If you need actual security for sensitive content, use password protection or server-level blocks. I’ve lost patience with advice that calls robots.txt a way to hide private pages, because listing a private directory in a public file hands its location to every scraper that reads it.
How Do Robots.txt Mistakes Happen?
Every character in a robots.txt file matters. The difference between blocking your admin folder and blocking your entire website is a single word:
Disallow: /wp-admin/ blocks your admin folder. Correct.
Disallow: / blocks your entire website. Catastrophic.
The book on this: The Day Your Website Died is forty-two chapters on how answer engines decide who gets named, and what to do about it.
Other common mistakes: blocking /blog/ when you meant to block /blog/drafts/, forgetting that paths are case sensitive, and deploying a development robots.txt file to your live site. Each can remove pages or whole sections from search results, and recovery takes far longer than the mistake took to make. That last one makes me angriest, because the site owner eats the traffic loss for a launch process somebody else built without a single check in it.
What does a safe robots.txt file look like?
Basic WordPress Site
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap.xmlThis blocks WordPress admin areas that shouldn’t be indexed while allowing a necessary AJAX file. Conservative and safe. Start here.
Small Business Site
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /temp/
Sitemap: https://yourbusiness.com/sitemap.xmlProtects admin areas and private content while leaving all public pages accessible.
E-commerce Site
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search?
Sitemap: https://yourstore.com/sitemap.xmlBlocks transactional pages that create duplicate content issues while protecting customer data.
Development Site (Temporary)
User-agent: *
Disallow: /
Sitemap: https://devsite.com/sitemap.xmlBlocks everything. Use only on development sites. Remove immediately when going live. This is the line that will destroy your search visibility if it accidentally ends up on your production site.
Advanced Techniques
Targeting Specific Search Engines
User-agent: Googlebot
Disallow: /premium-content/
User-agent: AhrefsBot
Disallow: /
User-agent: *
Disallow: /admin/Different rules for different crawlers. This example blocks aggressive SEO tool crawlers entirely while allowing Google access to most content.
Pattern Blocking with Wildcards
User-agent: *
Disallow: /*?print=
Disallow: /*&utm_source=
Disallow: /category/*/page/Blocks URL patterns that create duplicate content or waste crawl budget without blocking individual pages.
How Do You Control AI Crawlers With Robots.txt?
Robots.txt is evolving to address AI-powered bots that scrape content for training language models and generating AI answers. This is a rapidly changing area, but there are already practical directives you can use.
Google and Apple have introduced specific user agents for their AI training crawlers, separate from their search crawlers. This means you can block AI training without affecting your search visibility:
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /These directives tell Google and Apple not to use your content for AI model training while still allowing their regular search crawlers to index your site normally.
Cloudflare has expanded robots.txt further with Content-Signal directives that provide granular control over how AI bots use your content. These include search=yes/no to control whether content can build search indexes, ai-input=yes/no to determine if content can be used as input for AI-generated answers like chatbot responses, and ai-train=yes/no to block or allow content for training AI models. The defaults are search=yes, ai-train=no, and ai-input=neutral until the site owner sets them.
Emerging standards like llms.txt and tools like Spawning’s Don’t Train protocol offer additional layers of control at the content level instead of the site level.
The critical caveat: no part of robots.txt binds every bot. That applies to every directive in the file, AI-specific or otherwise, and AI companies may ignore them entirely. These directives work mostly as legal reservations of rights and best-effort signals, so combine them with firewalls and access controls if you’re serious about preventing AI scraping. The burden should never fall on site owners at all. The companies scraping the content should have to ask first, and too many of them have decided not to bother. (See also how Reddit is answering the scrapers with a login wall)
| Directive or tool | What it controls |
|---|---|
| User-agent: Google-Extended | Blocks Google from using your content for AI training; search crawling continues |
| User-agent: Applebot-Extended | Blocks Apple from using your content for AI training; search crawling continues |
| Content-Signal search=yes/no | Whether content can build search indexes (default yes) |
| Content-Signal ai-input=yes/no | Whether content can feed AI-generated answers (neutral until set) |
| Content-Signal ai-train=yes/no | Whether content can train AI models (default no) |
| llms.txt and Spawning’s Don’t Train | Control at the content level instead of the site level |
| Firewalls and access controls | The only real barrier, since no directive binds every bot |
How do you test a robots.txt file?
Never upload a robots.txt file without testing it first. Use the robots.txt report in Google Search Console to check syntax and verify that your rules do what you intend. Test multiple URLs, including your homepage, your most important content pages, and the pages you’re trying to block. Verify that your sitemap location is correct.
Back up your current robots.txt file before making changes. Start with minimal restrictions and add rules gradually. Upload during low-traffic hours. Monitor crawl errors and traffic data for at least 48 hours after any change. If something looks wrong, revert to your backup immediately.
When Something Goes Wrong
If you discover a robots.txt error that’s blocking content from search engines, fix the file immediately. Then request a re-crawl in Google Search Console and resubmit your sitemap. Monitor crawl errors daily for the first week. Check the indexing status of pages that were blocked. Submit individual URLs for re-indexing if necessary.
Recovery time depends on how long the error was live and how much content was affected. A few hours of blocking might resolve within days. Weeks of blocking can take months to fully recover from. The longer bad directives stay in place, the more damage they cause and the longer the recovery takes. For more, see protecting your site with backups.
Google has finally put rough numbers on the wait. At Search Central Live in Barcelona on October 2, 2026, Gary Illyes walked through typical and worst-case timings, and Search Engine Journal reported them from an attendee’s recap.
A page Google already knows about typically gets recrawled in about 30 days. A sitemap is usually processed within a day but can take up to two weeks. Indexing itself takes about an hour and a half once Google gets around to the page.
The slow end of those ranges runs to months, and sometimes to never. So after you fix the file and resubmit the sitemap, give it a month before you panic, and expect the pages Google rarely visited to come back last. I went through the full set of numbers in how long Google takes to crawl, index and recover.
WordPress Specifics
WordPress automatically generates a basic robots.txt file with minimal protection for admin areas.
This default is safe but basic. Most serious WordPress sites benefit from a customized file. My WordPress help for writers exists for exactly this. SEO plugins like SEOpress let you edit robots.txt directly from your admin dashboard. That makes changes easier but also easier to break.
Don’t block /wp-content/uploads/ where your images are stored. Blocking this directory kills your image SEO. Do block /wp-admin/ and /wp-includes/. Always allow /wp-admin/admin-ajax.php which WordPress needs for core functionality.
My take
Leave robots.txt alone unless you need to block something specific, and test every change before it goes live. One wrong line can pull your whole site out of Google. Back up the current file, check your key URLs in Search Console, and watch crawl data for two days after. If you break it, fix it, resubmit the sitemap and give it a month before you panic. Don’t block your uploads folder.
The Point
Most successful websites use conservative robots.txt files that protect obvious admin areas and leave everything else accessible. The goal is a file that safely serves your business without risking the kind of mistake that removes your site from search results.
Start simple, test everything, and watch your traffic after every change. When in doubt, allow access and accept a little wasted crawling over the risk of blocking important content. I’d take a boring robots.txt file over a clever one every time, because the clever ones are the files that wipe out a business’s search traffic while nobody’s watching, and the owner is the one who pays for it.
Robots.txt Frequently Asked Questions
One wrong line can hide a site from search, and AI search changed the rules again. The Day Your Website Died is my book on optimizing for that new kind of traffic.
- Google Search Data On Crawling, Indexing and Serving Timelines, Search Engine Roundtable
- Giving users choice with Cloudflare’s new Content Signals Policy, Cloudflare
- Google Clarifies the “Google-Extended” Crawler Documentation, Search Engine Journal
