sharetrader
Results 1 to 2 of 2
  1. #1
    Guru drillfix's Avatar
    Join Date
    May 2008
    Posts
    4,713

    Default Baidu Spider Bot in droves

    Hi vince, admins and mods,

    Just wondering if anybody notices the amount of spider bots on the ST pages when evening hits (or day for that matter).

    Sometimes there are nearly 40 - 50 instances of this spider bot and from reading about it, it seems chinese and does not seem to either understand or obey any Robots.txt file

    Perhaps a

    User-agent: Baiduspider
    Disallow: /*

    or something similar but then, I dont know how much of a pest it is as usually bots come and go yet not in Army formation as the Baidu one does.

    Just reporting this instance and a heads up to the web folks here at ShareTrader.

    Cheers~!
    --> Watch THRIVE 2011 a must watch Doco! --> Also watch ETHOS 2011 & share the Awareness~!

  2. #2
    Administrator
    Join Date
    Feb 2010
    Posts
    25

    Default

    Hello, yes, the site is spidered all the time. But where do you notice the effect of Baidu on ST web pages?

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •