Baidu Spider Bot in droves
Hi vince, admins and mods,
Just wondering if anybody notices the amount of spider bots on the ST pages when evening hits (or day for that matter).
Sometimes there are nearly 40 - 50 instances of this spider bot and from reading about it, it seems chinese and does not seem to either understand or obey any Robots.txt file
Perhaps a
User-agent: Baiduspider
Disallow: /*
or something similar but then, I dont know how much of a pest it is as usually bots come and go yet not in Army formation as the Baidu one does.
Just reporting this instance and a heads up to the web folks here at ShareTrader.
Cheers~!