PDA

View Full Version : Baidu Spider Bot in droves



drillfix
12-10-2011, 03:24 AM
Hi vince, admins and mods,

Just wondering if anybody notices the amount of spider bots on the ST pages when evening hits (or day for that matter).

Sometimes there are nearly 40 - 50 instances of this spider bot and from reading about it, it seems chinese and does not seem to either understand or obey any Robots.txt file

Perhaps a

User-agent: Baiduspider
Disallow: /*

or something similar but then, I dont know how much of a pest it is as usually bots come and go yet not in Army formation as the Baidu one does.

Just reporting this instance and a heads up to the web folks here at ShareTrader.

Cheers~!

Trader_Tech
11-11-2011, 08:05 AM
Hello, yes, the site is spidered all the time. But where do you notice the effect of Baidu on ST web pages?