When robots (like the Googlebot) crawl your site, they begin by requesting
http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and edit your robots.txt file from within WordPress (using
Options -> Robots.txt).
Whenever a user (or a robot, more likely) appends “robots.txt” to your blog URL (e.g. http://blog.example.com/robots.txt), this plugin will serve up the robots.txt file that you created in the WordPress admin menu.
This plugin should work with most versions of WordPress, but it is particularly intended for WP-MU installations, since it allows each WPMU blog to have a unique robots.txt file.
Download this plugin from the wordpress.org repository .
Note that robots make only top-level requests for robots.txt files. If you have WordPress installed in a subdomain or in your root, this plugin will work as intended. But if you have WordPress installed in a subdirectory, then this plugin won’t do much for you, since the search engines won’t look for http://example.com/blog/robots.txt, only for http://example.com/robots.txt. Just to make sure this is clear,
- WordPress in a subdomain: http://blog.example.com/
- WordPress at the root: http://example.com/
- WordPress in a subdirectory: http://example.com/blog/
If the FAQs at the download page don’t answer your questions, you can post support questions below.