drillfix
12-10-2011, 04:24 AM
Hi vince, admins and mods,
Just wondering if anybody notices the amount of spider bots on the ST pages when evening hits (or day for that matter).
Sometimes there are nearly 40 - 50 instances of this spider bot and from reading about it, it seems chinese and does not seem to either understand or obey any Robots.txt file
Perhaps a
User-agent: Baiduspider
Disallow: /*
or something similar but then, I dont know how much of a pest it is as usually bots come and go yet not in Army formation as the Baidu one does.
Just reporting this instance and a heads up to the web folks here at ShareTrader.
Cheers~!
Just wondering if anybody notices the amount of spider bots on the ST pages when evening hits (or day for that matter).
Sometimes there are nearly 40 - 50 instances of this spider bot and from reading about it, it seems chinese and does not seem to either understand or obey any Robots.txt file
Perhaps a
User-agent: Baiduspider
Disallow: /*
or something similar but then, I dont know how much of a pest it is as usually bots come and go yet not in Army formation as the Baidu one does.
Just reporting this instance and a heads up to the web folks here at ShareTrader.
Cheers~!