+- Southperry.net (https://www.southperry.net)
+-- Forum: Main (https://www.southperry.net/forumdisplay.php?fid=11)
+--- Forum: Site Announcements (https://www.southperry.net/forumdisplay.php?fid=18)
+--- Thread: Instability Mystery Solved (/showthread.php?tid=49245)
Instability Mystery Solved - Eos - 2011-12-01
So, I just found the reason for our random periods where the site starts to grind to a halt when usage is normal...
Apparently certain search engines that will not be named when spidering us, do in fact run a 'show all posts by this user' on every post.
An explicit "do not do this" rule has been added to our robots.txt and the offending spiders have been blocked at the firewall for the next 24 hours until they have time to notice (and hopefully obey) the rules.
Instability Mystery Solved - Locked - 2011-12-01
Damn you..
...Bing
..Yahoo!
..Dogpile
..Brainboost
...YaCy
..WebCrawler
etc.
Instability Mystery Solved - MuscleWizard - 2011-12-01
Good Eye.
Instability Mystery Solved - Corn - 2011-12-01
Oh, that explains a lot of things.
Instability Mystery Solved - Eos - 2011-12-01
Locked Wrote:etc.
Except for MSN most of them weren't even US.
Instability Mystery Solved - iamflip - 2011-12-01
Eos Wrote:Except for MSN most of them weren't even US.
Damn those Chinese spider crawlers.
Instability Mystery Solved - Chameleonic - 2011-12-05
They still misbehaving? Cause SP is running so slow right now.
Instability Mystery Solved - Justin - 2011-12-05
SP was just flatout unreachable for me for a little while there... I'm assuming your "solution" didn't slow them down any
Instability Mystery Solved - Eos - 2011-12-05
Rather certain ones of them are blatantly choosing to ignore their own stated guidelines.
And since they're assinine enough to use dozens of different IP addresses the only fix effectively blocks entire subnets.
Instability Mystery Solved - Eos - 2011-12-05
Ugh, look at this crap:
June's logs for this one search agent: (number after colon is how many hits from specific bot that day)
Spoiler
November's logs:
Spoiler
And we know damn well Southperry's traffic hasn't picked up that significantly.
I've firewalled it, robots.txt'd it, and sent en email to it's support address with the 8mb of logs from the past 72 hours of just it re requesting the same threads once per post.
Instability Mystery Solved - Justin - 2011-12-05
An average of ten THOUSAND hits a day? Sweet jesus.
Instability Mystery Solved - Chameleonic - 2011-12-05
Justin Wrote:An average of ten THOUSAND hits a day? Sweet jesus.
And thats just from one search agent...no wonder things grind to a shuddering halt.
Instability Mystery Solved - MuscleWizard - 2011-12-05
Holy fucking pomegranate eos.
that's a lot of traffic
Instability Mystery Solved - Shidoshi - 2011-12-05
Could it be some kind of attempt at DoS?
Instability Mystery Solved - Fiel - 2011-12-05
Shidoshi Wrote:Could it be some kind of attempt at DoS?
A search spider that is also a DoS is counterintuitive, counterproductive, and just plain stupid.
A search spider that tries to bring other people to Southperry when the spider itself is making the site unreachable... that's a logic clusterf'uck.