So, I just found the reason for our random periods where the site starts to grind to a halt when usage is normal...
Apparently certain search engines that will not be named when spidering us, do in fact run a 'show all posts by this user' on every post.
An explicit "do not do this" rule has been added to our robots.txt and the offending spiders have been blocked at the firewall for the next 24 hours until they have time to notice (and hopefully obey) the rules.
Bookmarks