Printable View
So, I just found the reason for our random periods where the site starts to grind to a halt when usage is normal...
Apparently certain search engines that will not be named when spidering us, do in fact run a 'show all posts by this user' on every post.
An explicit "do not do this" rule has been added to our robots.txt and the offending spiders have been blocked at the firewall for the next 24 hours until they have time to notice (and hopefully obey) the rules.
Damn you..
...Bing
..Yahoo!
..Dogpile
..Brainboost
...YaCy
..WebCrawler
etc.
Good Eye.
Oh, that explains a lot of things.
Quote:
Originally Posted by Locked
Except for MSN most of them weren't even US.
Damn those Chinese spider crawlers. :f3:Quote:
Originally Posted by Eos
They still misbehaving? Cause SP is running so slow right now.
SP was just flatout unreachable for me for a little while there... I'm assuming your "solution" didn't slow them down any :(
Rather certain ones of them are blatantly choosing to ignore their own stated guidelines.
And since they're assinine enough to use dozens of different IP addresses the only fix effectively blocks entire subnets.
Ugh, look at this crap:
June's logs for this one search agent: (number after colon is how many hits from specific bot that day)
Spoiler
November's logs:
Spoiler
And we know damn well Southperry's traffic hasn't picked up that significantly.
I've firewalled it, robots.txt'd it, and sent en email to it's support address with the 8mb of logs from the past 72 hours of just it re requesting the same threads once per post. :f7:
An average of ten THOUSAND hits a day? Sweet jesus.
And thats just from one search agent...no wonder things grind to a shuddering halt.Quote:
Originally Posted by Justin
Holy pineappleing pomegranate eos.
that's a lot of traffic
Could it be some kind of attempt at DoS?
A search spider that is also a DoS is counterintuitive, counterproductive, and just plain stupid.Quote:
Originally Posted by Shidoshi
A search spider that tries to bring other people to Southperry when the spider itself is making the site unreachable... that's a logic clusterf'uck.