Results 1 to 15 of 15
  1. Default Instability Mystery Solved

    So, I just found the reason for our random periods where the site starts to grind to a halt when usage is normal...

    Apparently certain search engines that will not be named when spidering us, do in fact run a 'show all posts by this user' on every post.

    An explicit "do not do this" rule has been added to our robots.txt and the offending spiders have been blocked at the firewall for the next 24 hours until they have time to notice (and hopefully obey) the rules.

  2. Default

    Damn you..



  3. aka ClawofBeta Straight Male
    Corn's Avatar [Jr. Event Coordinator]

    IGN: ClawofBeta
    Server: LoL.NA
    Level: 30
    Job: Bot Lane
    Guild: N/A
    Alliance: N/A


    Oh, that explains a lot of things.

  4. Default

    Except for MSN most of them weren't even US.

  5. Default

    Damn those Chinese spider crawlers.

  6. Default

    SP was just flatout unreachable for me for a little while there... I'm assuming your "solution" didn't slow them down any :(

  7. Default

    Rather certain ones of them are blatantly choosing to ignore their own stated guidelines.

    And since they're assinine enough to use dozens of different IP addresses the only fix effectively blocks entire subnets.

  8. Default

    Ugh, look at this crap:

    June's logs for this one search agent: (number after colon is how many hits from specific bot that day)

    November's logs:

    And we know damn well Southperry's traffic hasn't picked up that significantly.
    I've firewalled it, robots.txt'd it, and sent en email to it's support address with the 8mb of logs from the past 72 hours of just it re requesting the same threads once per post.

  9. Default

    An average of ten THOUSAND hits a day? Sweet jesus.

  10. Default

    And thats just from one search wonder things grind to a shuddering halt.

  11. Default

    Holy pineappleing pomegranate eos.
    that's a lot of traffic

  12. Default

    A search spider that is also a DoS is counterintuitive, counterproductive, and just plain stupid.

    A search spider that tries to bring other people to Southperry when the spider itself is making the site unreachable... that's a logic clusterf'uck.



Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts