Administrator
Posts: 17,190
Threads: 2,153
Joined: 2008-09
Gender: Male
Sexual Orientation: Gay
IGN: Aesilyn
Server: Mardia
Level: 200
Job: I/L ArchMage
Guild: Animus
So, I just found the reason for our random periods where the site starts to grind to a halt when usage is normal...
Apparently certain search engines that will not be named when spidering us, do in fact run a 'show all posts by this user' on every post.
An explicit "do not do this" rule has been added to our robots.txt and the offending spiders have been blocked at the firewall for the next 24 hours until they have time to notice (and hopefully obey) the rules.
It's not having what you want - It's wanting what you've got.
Posting Freak
Posts: 9,907
Threads: 379
Joined: 2010-02
Damn you..
..Google
...Bing
..Yahoo!
..Dogpile
..Brainboost
...YaCy
..WebCrawler
etc.
Posting Freak
Posts: 13,609
Threads: 249
Joined: 2008-07
Gender: Male
Sexual Orientation: Straight
Country Flag: New_Jersey
IGN: ClawofBeta
Server: LoL.NA
Level: 30
Job: Bot Lane
Guild: N/A
Guild Alliance: N/A
Oh, that explains a lot of things.
Administrator
Posts: 17,190
Threads: 2,153
Joined: 2008-09
Gender: Male
Sexual Orientation: Gay
IGN: Aesilyn
Server: Mardia
Level: 200
Job: I/L ArchMage
Guild: Animus
Locked Wrote:etc.
Except for MSN most of them weren't even US.
It's not having what you want - It's wanting what you've got.
Posting Freak
Posts: 4,490
Threads: 167
Joined: 2008-07
They still misbehaving? Cause SP is running so slow right now.
Administrator
Posts: 17,190
Threads: 2,153
Joined: 2008-09
Gender: Male
Sexual Orientation: Gay
IGN: Aesilyn
Server: Mardia
Level: 200
Job: I/L ArchMage
Guild: Animus
Rather certain ones of them are blatantly choosing to ignore their own stated guidelines.
And since they're assinine enough to use dozens of different IP addresses the only fix effectively blocks entire subnets.
It's not having what you want - It's wanting what you've got.
Administrator
Posts: 17,190
Threads: 2,153
Joined: 2008-09
Gender: Male
Sexual Orientation: Gay
IGN: Aesilyn
Server: Mardia
Level: 200
Job: I/L ArchMage
Guild: Animus
Ugh, look at this crap:
June's logs for this one search agent: (number after colon is how many hits from specific bot that day)
November's logs:
And we know damn well Southperry's traffic hasn't picked up
that significantly.
I've firewalled it, robots.txt'd it, and sent en email to it's support address with the 8mb of logs from the past 72 hours of just it re requesting the same threads once per post.
It's not having what you want - It's wanting what you've got.
Posting Freak
Posts: 4,710
Threads: 74
Joined: 2008-07
An average of ten THOUSAND hits a day? Sweet jesus.
Posting Freak
Posts: 3,524
Threads: 151
Joined: 2011-01
Gender: Male
Sexual Orientation: Bi
Country Flag: New_Jersey
IGN: Rebootycall
Server: Reboot
Level: 173
Job: Wild Hunter
Holy fucking pomegranate eos.
that's a lot of traffic
Posting Freak
Posts: 6,052
Threads: 182
Joined: 2008-07
Could it be some kind of attempt at DoS?
