-
Baidu Spider Bot in droves
Hi vince, admins and mods,
Just wondering if anybody notices the amount of spider bots on the ST pages when evening hits (or day for that matter).
Sometimes there are nearly 40 - 50 instances of this spider bot and from reading about it, it seems chinese and does not seem to either understand or obey any Robots.txt file
Perhaps a
User-agent: Baiduspider
Disallow: /*
or something similar but then, I dont know how much of a pest it is as usually bots come and go yet not in Army formation as the Baidu one does.
Just reporting this instance and a heads up to the web folks here at ShareTrader.
Cheers~!
-
Hello, yes, the site is spidered all the time. But where do you notice the effect of Baidu on ST web pages?
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
|
Bookmarks