Introduction
Banning spiders and agents
Search engine spider identification
Further learning resources:
If you've been surfing search engine optimization web sites, you've no doubt come across the term spider, robot or bot on many occasions.
Five terms all describing basically the same thing, but in this article they'll be referred to collectively as spiders or "agents". A search engine spider is an automated software program used to locate and collect data from web pages for inclusion in a search engine's database and to follow links to find new pages on the World Wide Web. The term "agent" is more commonly applied to web browsers and mirroring software.
If you've ever examined your server logs or web site traffic reports, you've probably come across some weird and wonderful names for search engine spiders, including "Fluffy the Spider" and Slurp. Depending upon the type of web traffic reports you receive, you may find spiders listed in the "Agents" section of your statistics.
Who actually owns these spiders? It's good to know the beneficial from the bad. Some agents are generated by software such as Teleport Pro, an application that allows people to download a full "mirror" of your site onto their hard drives for viewing later on, or sometimes for more insidious purposes such as plagiarism. If you have a large or image heavy site, the practice of web site stripping could also have a serious impact on your bandwidth usage each month.
Michael Bloch
www.tamingthebeast.netMichael Bloch
Taming the Beast.net
http://www.tamingthebeast.ne...
Tutorials, web content, tools and software.
Web Marketing, eCommerce & Development solutions.
_____________________________________________
Copyright information.... This article is free for reproduction but must be reproduced in its entirety & this copyright statement must be included. Visit http://www.tamingthebeast.ne... to view great articles, tutorials and tools for site owners, web developers and Internet marketers! Subscribe for free to our popular ecommerce/web design ezine!