When robots (like the Googlebot) crawl your site, they begin by requesting http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within WordPress (using Options -> Robots.txt).
Whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the WordPress admin menu.
This plugin should work with most versions of WordPress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file.
Download this plugin from the wordpress.org repository 
.
Limitations
Note that robots make only top-level requests for robots.txt files. If you have WordPress installed in a subdomain or in your root, this plugin will work as intended. But if you have WordPress installed in a subdirectory, then this plugin won’t do much for you, since the search engines won’t look for http://example.com/blog/robots.txt, only for http://example.com/robots.txt. Just to make sure this is clear,
- WordPress in a subdomain: http://blog.example.com/
- WordPress at the root: http://example.com/
- WordPress in a subdirectory: http://example.com/blog/
Support
If the FAQs at the download page don’t answer your questions, you can post support questions below.
30 Comments
Adam,
Brilliant plug in, placed a note about it in WPMU forum. WPMU doesn’t behave exactly intuitively at the blogs.dir level, and this plugin neatly sidesteps the whole issue to meet a very important requirement for subdomains and full external domains running off WPMU.
Have you considered serving a Google Sitemap using the same strategy?
Thanks.
I’m not sure about the Google Sitemap thing. There would need to be some sort of caching mechanism, or else large blogs would take a very long time to generate a sitemap each time. I guess people could write a sitemap by hand, but that’s not so fun.
The caching could be achieved by generating the sitemap after new content is created, and storing the output in wp options – equivalent to the config text you store for robots. So when Google requests sitemap.xml, you just serve the text straight out of the table.
Maybe there is a way to queue a long running job – e.g. after updating content, launch the sitemap generation in batch mode? – that would avoid any performance hit for the author.
There is a 10Mb limit on file size – while a compressed version doesn’t seem to be mandatory, producing and storing a .gz version to support very large blogs may present issues.
https://www.google.com/webmasters/tools/docs/en/protocol.html
The reason I suggested this is that – as I read it from the wpmu forum – getting sitemaps working in WPMU means installing the plugin, pre-populating the sitemap.xml and sitemap.xml.gz (to avoid permission errors), modifying .htaccess, and modifying allowed files. Messy. Your dynamic generation solution would avoid the .htaccess mods and physical file altogether, which for WPMU installations makes a *lot* of sense.
Possibly Arne Brachold’s code (for generating the sitemap itself) would be reusable – so it might just be a matter of wrappering Arne’s map generation code within your page serving model?
I see. Not a bad idea, but I probably won’t have time to do it.
I installed this plugin and changed the robots.txt file through the interface, but nothing has changed in the actual file itself. What am I missing here? I’m not using WP-MU, just the regular WordPress.org install.
-Alex
I’m not sure what you mean, Alex. There isn’t any file involved here. The robots.txt gets served from your blog’s database, just like your posts do. The plugin doesn’t create or edit any files.
Incidentally, if you do have a file called robots.txt installed in your blog’s directory, then you might need to delete it for this plugin to work correctly (depending on your htaccess settings).
Thanks for the speedy reply! I’m not sure what the problem was either. I tried deleting the robots.txt file that was already there, but when I entered my changes through KB Robots in the WordPress interface and clicked “View Robots.txt” after save the changes, none of them were reflected in what I saw. I ended manually creating the robots.txt and putting it back in the blog directory.
-Alex
Hi Adam,
I installed your plugin. This what I entered in the robots.txt plugin window.
User-agent: *
Disallow:
But when I do I do [link] this is what I get, and in the plugin when it says check the the robots.txt file after I submit I get this also:
# BEGIN XML-SITEMAP-PLUGIN
Sitemap: [link]
# END XML-SITEMAP-PLUGIN
So uninstalled the google sitemap and analytics, and still the same thing. I was wondering could you help me solve this problem.
Thanks,
Stacey
I fixed it. I wasnt putting it in the root directory.
Hmm. It doesn’t go in the root, it goes in the plugins folder, but if you’ve got it working, great.
Hi
I am using WP-MU. and installed your plugin. and pasted the wordpress robot text from your one of the post. but now when i want to see the robot.txt then inplace to going to [link] it goes to
[link] URl
Please help me and tell me what to do..
I don’t know what to say. I use it on WPMU without problems. One thing I haven’t tried is using it on the root WPMU blog–I’ve only tried it on member blogs. Maybe that’s the problem. Or, it could be that your version of WPMU doesn’t support the action hook that the plugin uses. I think it’s the
inithook, but I don’t remember.Regardless, I don’t have the time to look into this at the moment, and probably won’t for several weeks. But if you’re running WPMU, you are probably familiar enough with PHP and WP to debug this fairly quickly. If you find a bug, let me know and I’ll patch the plugin. Good luck.
Hi Adam,
I am writing to ask a question about your WP MU robots.txt plugin “KB Robots.txt”
It sounds great and thanks for creating it.
I am new to using WP MU and just installed it on a site of mine however, I already have a WP blog on my main domain so WP MU is in a sub-directory.
Therefore, it’s like [link] (where blogs is my WP MU install)
Since you say, “But if you have WordPress installed in a subdirectory, then this plugin won’t do much for you, since the search engines won’t look for [link]” — does this mean it’s useless for me to install your plugin as well as even having a robots.txt file with WP MU installed in a sub-directory?
Even though I have a robots.txt for my root domain could I not somehow add robot information for my WP MU?
For example, say right now in my main root folder [link] I have a robots.txt file with User-agent: *
Allow: /
Could I not simply amend this and also put Allow: /blogs/ ?
This way my robots.txt file is in my main root folder (public_html) and the SE’s can hopefully follow on to the MU blogs??
If the answer is “yes” then should I duplicate all(most) of the instructions in my robots.txt file but add /blogs/ to each? Does that make sense? So if I had User-agent: * Allow: / I would also have Allow: /blogs/ for command?
I apologize for this long email. Just wondering if I can apply robot info to my WP MU in my sub-directory somehow :)
Thanks again
Putting a robots.txt file into a subdirectory will not accomplish anything, so, no, the plugin won’t help you. More info:
[link]
Adam, from the comments above it sounds as though this is a must have plugin. I just don’t fully understand the benefits I’ll get from it though. Is there any other info I can read to understand it better? Sorry to have to ask, I’m sure you’re very busy.
Hi – I need to create the robot.txt file to exclude the robots just from specific pages on the wp site rather than whole areas – like the archives. I looked at the examples you have in the plugin – I just need some clarification on how to exclude specific pages.
Thanks
Hi.
Thank you for KB Robots.txt wordpress plugin. It works very well!
I just wanted to let you know that there is a bug:
If /wp-admin/ requires https \ ssl then firefox will complain that the submit button will transfer data in a non secure way, etc. and that is because http:// is hard-coded in kb-robots-txt.php line 59.
The fix for me was just to replace http:// with https:// however I would like the plugin to dynamically determine this from checking the php $_SERVER['HTTPS'] variable for example.
Thank you.
Just reporting that the plugin works flawlessly on WPMU 2.7 with subdomains – very nice job. Thanks for putting it together.
Hi Adam,
Just a quick note to say thanks for this.
-drmike
I’ve installed the plug-in on WP 2.7.1 and I would like to think that I’m of average intelligence but I can’t seem to figure out how to use this. I’ve went to [link] and it returns a 404 not found. I know you’re a busy man and I am checking the plug-ins documentation, thanks in advance.
I was looking for a robots.txt manager plugin and looked at your limitations – and I think you are over limiting your plugin. I use wordpress in a subdirectory named wordpress. To get the robots.txt you generate to be used I tried a symbolic link in the document root – that did not work. So I added a rewrite rule to my document root .htaccess file:
# use wordpress robots.txt file<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^robots\.txt$ /wordpress/robots.txt
</IfModule>
This is working great at this time. I am not a security expert – so if anyone sees a security issue with this, please let me know.
Great plugin – thanks for creating it and sharing it with everyone.
this is brilliant! thanks so much for sharing
Hey I’m confused. Where does it create the robot.txt cause i don’t see it in my roots directly…i’m also using wp-hive and so if i know how to manually create the robot.txt file i can move it to the directory that i need it to be in plz email me back
This plugin doesn’t work with most of the now WP wersions, that’s why Adam is not responding on your massages.
[immature profanity removed by site owner]
@Peter: Actually, if somebody offered a “massage,” I might make responding a higher priority. I generally write plugins for my own use, share them on wp.org, then don’t tend to them further unless I have a need to. I don’t actually use this plugin anymore, hence minimal attention to it. That being said, it’s a very simple plugin that should work on any version of WP unless they changed something drastic in the code.
@Jon: It does not create a robots.txt file. Just like how WP doesn’t create an HTML file for each blog post.
@Ken C: Shouldn’t require any changes to htaccess for this plugin to work, as long as you’ve got WP installed in the root, e.g. example.com, not example.com/blog/.
just installed this on a fresh WP 3.0RC1 install
one thing i noticed is that it will not work unless you have triggered a change in the settings that generates an .htaccess file, e.g. changed your permalinks settings.
then it works brilliantly
I am trying to have one of my pages NOT be indexed. I used the Ultimate Noindex Nofollow tool plugin – but that is not working – the page numbers I specify are getting indexed anyway. Someone suggested I use a Robots.txt file instead.
since I am not sure how to do that, I looked through the forum and found KB Robots.txt plugin.
I’m wondering – if i turn OFF (uncheck) the option in the Google XML sitemaps plugin to have the URL be placed in a virtual robots.txt file, can I then also use KB Robots to generate a robots.txt file with the pages I don’t want indexed disallowed?
Not sure why the Ultimate plugin is not working, but I am looking for a work around
thanks
The instructions say, “Use this plugin to create and edit your robots.txt file from within WordPress (using Options -> Robots.txt).”
Someone please tell me where in the world I might find this “Options -> Robots.txt”.
The only Options menu item I see is in the Super Admin Menu (of the blog network), and that leads to a Network Options page which has:
Operational Settings,
Dashboard Settings,
Registration Settings,
New Site Settings,
Upload Settings, &
Menu Settings.
But no Robots.txt editing area.
What’s the deal, am I blind?
@tunecrew: Yes, as stated in the instructions, it won’t work unless you’re using permalinks.
@A: I’m not familiar with the other plugin that you’re asking about.
@Terry: First off, relax. Second, remember that you’re using a very old plugin that hasn’t been updated in a long time, nor will it be anytime soon. The WP developers have renamed the “Options” menu to the “Settings” menu. Look there.
Great plugin, but you should have posted the default (kind of) robots.txt rules set for wordpress blogs with the post.