Adam R Brown

Adam,
Brilliant plug in, placed a note about it in WPMU forum. WPMU doesn’t behave exactly intuitively at the blogs.dir level, and this plugin neatly sidesteps the whole issue to meet a very important requirement for subdomains and full external domains running off WPMU.
Have you considered serving a Google Sitemap using the same strategy?

The caching could be achieved by generating the sitemap after new content is created, and storing the output in wp options - equivalent to the config text you store for robots. So when Google requests sitemap.xml, you just serve the text straight out of the table.

Maybe there is a way to queue a long running job - e.g. after updating content, launch the sitemap generation in batch mode? - that would avoid any performance hit for the author.

There is a 10Mb limit on file size - while a compressed version doesn’t seem to be mandatory, producing and storing a .gz version to support very large blogs may present issues.
https://www.google.com/webmasters/tools/docs/en/protocol.html

The reason I suggested this is that - as I read it from the wpmu forum - getting sitemaps working in WPMU means installing the plugin, pre-populating the sitemap.xml and sitemap.xml.gz (to avoid permission errors), modifying .htaccess, and modifying allowed files. Messy. Your dynamic generation solution would avoid the .htaccess mods and physical file altogether, which for WPMU installations makes a *lot* of sense.

Possibly Arne Brachold’s code (for generating the sitemap itself) would be reusable - so it might just be a matter of wrappering Arne’s map generation code within your page serving model?

I installed this plugin and changed the robots.txt file through the interface, but nothing has changed in the actual file itself. What am I missing here? I’m not using WP-MU, just the regular Wordpress.org install.

-Alex

Thanks for the speedy reply! I’m not sure what the problem was either. I tried deleting the robots.txt file that was already there, but when I entered my changes through KB Robots in the Wordpress interface and clicked “View Robots.txt” after save the changes, none of them were reflected in what I saw. I ended manually creating the robots.txt and putting it back in the blog directory.

-Alex

I fixed it. I wasnt putting it in the root directory.

Hi
I am using WP-MU. and installed your plugin. and pasted the wordpress robot text from your one of the post. but now when i want to see the robot.txt then inplace to going to [link] it goes to
[link] URl

Please help me and tell me what to do..

Hi Adam,

I am writing to ask a question about your WP MU robots.txt plugin “KB Robots.txt”

It sounds great and thanks for creating it.

I am new to using WP MU and just installed it on a site of mine however, I already have a WP blog on my main domain so WP MU is in a sub-directory.

Therefore, it’s like [link] (where blogs is my WP MU install)

Since you say, “But if you have Wordpress installed in a subdirectory, then this plugin won’t do much for you, since the search engines won’t look for [link]” — does this mean it’s useless for me to install your plugin as well as even having a robots.txt file with WP MU installed in a sub-directory?

Even though I have a robots.txt for my root domain could I not somehow add robot information for my WP MU?

For example, say right now in my main root folder [link] I have a robots.txt file with User-agent: *
Allow: /

Could I not simply amend this and also put Allow: /blogs/ ?

This way my robots.txt file is in my main root folder (public_html) and the SE’s can hopefully follow on to the MU blogs??

If the answer is “yes” then should I duplicate all(most) of the instructions in my robots.txt file but add /blogs/ to each? Does that make sense? So if I had User-agent: * Allow: / I would also have Allow: /blogs/ for command?

I apologize for this long email. Just wondering if I can apply robot info to my WP MU in my sub-directory somehow :)

Thanks again

Adam, from the comments above it sounds as though this is a must have plugin. I just don’t fully understand the benefits I’ll get from it though. Is there any other info I can read to understand it better? Sorry to have to ask, I’m sure you’re very busy.

Hi.

Thank you for KB Robots.txt wordpress plugin. It works very well!

I just wanted to let you know that there is a bug:

If /wp-admin/ requires https \ ssl then firefox will complain that the submit button will transfer data in a non secure way, etc. and that is because http:// is hard-coded in kb-robots-txt.php line 59.

The fix for me was just to replace http:// with https:// however I would like the plugin to dynamically determine this from checking the php $_SERVER[’HTTPS’] variable for example.

Thank you.

Hi Adam,

Just a quick note to say thanks for this.

-drmike

I was looking for a robots.txt manager plugin and looked at your limitations - and I think you are over limiting your plugin. I use wordpress in a subdirectory named wordpress. To get the robots.txt you generate to be used I tried a symbolic link in the document root - that did not work. So I added a rewrite rule to my document root .htaccess file:
# use wordpress robots.txt file
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^robots\.txt$ /wordpress/robots.txt
</IfModule>
This is working great at this time. I am not a security expert - so if anyone sees a security issue with this, please let me know.

Great plugin - thanks for creating it and sharing it with everyone.