Results 1 to 15 of 15
  1. Default Instability Mystery Solved


    So, I just found the reason for our random periods where the site starts to grind to a halt when usage is normal...

    Apparently certain search engines that will not be named when spidering us, do in fact run a 'show all posts by this user' on every post.

    An explicit "do not do this" rule has been added to our robots.txt and the offending spiders have been blocked at the firewall for the next 24 hours until they have time to notice (and hopefully obey) the rules.

  2. Default


    Damn you..

    ..Google
    ...Bing
    ..Yahoo!
    ..Dogpile
    ..Brainboost
    ...YaCy
    ..WebCrawler


    etc.


  3. aka ClawofBeta Straight Male
    Corn's Avatar [Jr. Event Coordinator]

    IGN: ClawofBeta
    Server: LoL.NA
    Level: 30
    Job: Bot Lane
    Guild: N/A
    Alliance: N/A
    New_Jersey

    Default


    Oh, that explains a lot of things.

  4. Default



    Except for MSN most of them weren't even US.

  5. Default


    Damn those Chinese spider crawlers.


  6. Default


    SP was just flatout unreachable for me for a little while there... I'm assuming your "solution" didn't slow them down any :(

  7. Default


    Rather certain ones of them are blatantly choosing to ignore their own stated guidelines.

    And since they're assinine enough to use dozens of different IP addresses the only fix effectively blocks entire subnets.

  8. Default


    Ugh, look at this crap:

    June's logs for this one search agent: (number after colon is how many hits from specific bot that day)
    Spoiler


    November's logs:
    Spoiler


    And we know damn well Southperry's traffic hasn't picked up that significantly.
    I've firewalled it, robots.txt'd it, and sent en email to it's support address with the 8mb of logs from the past 72 hours of just it re requesting the same threads once per post.

  9. Default


    An average of ten THOUSAND hits a day? Sweet jesus.

  10. Default


    And thats just from one search agent...no wonder things grind to a shuddering halt.

  11. Default


    Holy pineappleing pomegranate eos.
    that's a lot of traffic


  12. Default


    A search spider that is also a DoS is counterintuitive, counterproductive, and just plain stupid.

    A search spider that tries to bring other people to Southperry when the spider itself is making the site unreachable... that's a logic clusterf'uck.

  13.  

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •