|
Strange
Dec 27, 2012 17:21:13 GMT -5
Post by D. N. Vercáriâ on Dec 27, 2012 17:21:13 GMT -5
A couple of minutes ago, a whole busload of lurkers seemed to storm through our peculiar den.
|
|
|
Strange
Dec 27, 2012 22:09:00 GMT -5
Post by andy on Dec 27, 2012 22:09:00 GMT -5
Maybe it's time for a robots.txt file in the root directory of the site, to keep the spiders away?
|
|
|
Strange
Dec 28, 2012 14:30:53 GMT -5
Post by D. N. Vercáriâ on Dec 28, 2012 14:30:53 GMT -5
We don't have access to any directory below the things that we're actually seeing, but never mind, spambots and data-mining bots are ignoring any settings and policies in robots.txt anyway.
|
|