When robots (like the Googlebot) crawl your site, they begin by requesting http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within Wordpress (using Options -> Robots.txt).
Whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the Wordpress admin menu.
This plugin should work with most versions of Wordpress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file.
Download this plugin from the wordpress.org repository 
.
Limitations
Note that robots make only top-level requests for robots.txt files. If you have Wordpress installed in a subdomain or in your root, this plugin will work as intended. But if you have Wordpress installed in a subdirectory, then this plugin won’t do much for you, since the search engines won’t look for http://example.com/blog/robots.txt, only for http://example.com/robots.txt. Just to make sure this is clear,
- Wordpress in a subdomain: http://blog.example.com/
- Wordpress at the root: http://example.com/
- Wordpress in a subdirectory: http://example.com/blog/
Support
If the FAQs at the download page don’t answer your questions, you can post support questions below.
12 comments »
Adam,
Brilliant plug in, placed a note about it in WPMU forum. WPMU doesn’t behave exactly intuitively at the blogs.dir level, and this plugin neatly sidesteps the whole issue to meet a very important requirement for subdomains and full external domains running off WPMU.
Have you considered serving a Google Sitemap using the same strategy?
Thanks.
I’m not sure about the Google Sitemap thing. There would need to be some sort of caching mechanism, or else large blogs would take a very long time to generate a sitemap each time. I guess people could write a sitemap by hand, but that’s not so fun.
The caching could be achieved by generating the sitemap after new content is created, and storing the output in wp options - equivalent to the config text you store for robots. So when Google requests sitemap.xml, you just serve the text straight out of the table.
Maybe there is a way to queue a long running job - e.g. after updating content, launch the sitemap generation in batch mode? - that would avoid any performance hit for the author.
There is a 10Mb limit on file size - while a compressed version doesn’t seem to be mandatory, producing and storing a .gz version to support very large blogs may present issues.
https://www.google.com/webmasters/tools/docs/en/protocol.html
The reason I suggested this is that - as I read it from the wpmu forum - getting sitemaps working in WPMU means installing the plugin, pre-populating the sitemap.xml and sitemap.xml.gz (to avoid permission errors), modifying .htaccess, and modifying allowed files. Messy. Your dynamic generation solution would avoid the .htaccess mods and physical file altogether, which for WPMU installations makes a *lot* of sense.
Possibly Arne Brachold’s code (for generating the sitemap itself) would be reusable - so it might just be a matter of wrappering Arne’s map generation code within your page serving model?
I see. Not a bad idea, but I probably won’t have time to do it.
I installed this plugin and changed the robots.txt file through the interface, but nothing has changed in the actual file itself. What am I missing here? I’m not using WP-MU, just the regular Wordpress.org install.
-Alex
I’m not sure what you mean, Alex. There isn’t any file involved here. The robots.txt gets served from your blog’s database, just like your posts do. The plugin doesn’t create or edit any files.
Incidentally, if you do have a file called robots.txt installed in your blog’s directory, then you might need to delete it for this plugin to work correctly (depending on your htaccess settings).
Thanks for the speedy reply! I’m not sure what the problem was either. I tried deleting the robots.txt file that was already there, but when I entered my changes through KB Robots in the Wordpress interface and clicked “View Robots.txt” after save the changes, none of them were reflected in what I saw. I ended manually creating the robots.txt and putting it back in the blog directory.
-Alex
Hi Adam,
I installed your plugin. This what I entered in the robots.txt plugin window.
User-agent: *
Disallow:
But when I do I do [link] this is what I get, and in the plugin when it says check the the robots.txt file after I submit I get this also:
# BEGIN XML-SITEMAP-PLUGIN
Sitemap: [link]
# END XML-SITEMAP-PLUGIN
So uninstalled the google sitemap and analytics, and still the same thing. I was wondering could you help me solve this problem.
Thanks,
Stacey
I fixed it. I wasnt putting it in the root directory.
Hmm. It doesn’t go in the root, it goes in the plugins folder, but if you’ve got it working, great.
Hi
I am using WP-MU. and installed your plugin. and pasted the wordpress robot text from your one of the post. but now when i want to see the robot.txt then inplace to going to [link] it goes to
[link] URl
Please help me and tell me what to do..
I don’t know what to say. I use it on WPMU without problems. One thing I haven’t tried is using it on the root WPMU blog–I’ve only tried it on member blogs. Maybe that’s the problem. Or, it could be that your version of WPMU doesn’t support the action hook that the plugin uses. I think it’s the
inithook, but I don’t remember.Regardless, I don’t have the time to look into this at the moment, and probably won’t for several weeks. But if you’re running WPMU, you are probably familiar enough with PHP and WP to debug this fairly quickly. If you find a bug, let me know and I’ll patch the plugin. Good luck.
Leave a comment