Internet Marketing Robot
|
|
Internet Marketing Robot |
     |
|
...easy-to-use, intelligent, internet robot that builds a link directory and creates link trades for you! |
 |
Robots
"Robots are coming but they only read my index.htm. In the last week, 5 or 6 have visited but none have analyzed my site."
Typical robot behavior is to visit the top-level page first.They will come back at intervals and follow the links from the index page: provided there are links, and the site doesn't use frames too heavily. They are programmed to space their requests out to avoid overloading any server with their activity. I took the time to track their course through a new multi-level site with about 50 pages by analyzing the log files for requests from known robot IP addresses. I only submitted the index.htm page, and had all the proper robot bait in place before I submitted.
Here's a composite picture:
First, there was a request for robots.txt, followed by a hit on the index.htm page. Within the week, there were requests for the pages linked from the index page, and they hit the third level pages by the end of the fortnight. Every visit was preceded by a request for robots.txt and about five pages (maximum) were requested each day.
Frames make it VERY difficult for robots and repeat visitors. Neither one can set bookmarks and jump back to where they want to go after leaving the site. Unless you have an absolutely overwhelming need for frames, they are more trouble than they are worth.
The Web Robots Database
http://info.webcrawler.com/mak/projects/robots/active.htmlSome of the entries are outdated,
but it remains a useful resource.Spider Spotting
http://searchenginewatch.com/spiders.htmWithin this site, it explains how
to track down robot visitors.
Main Search Engines Spiders and Robots ID Chart
Search Engine Agent Name Host Names googlebot google.com AltaVista
(regular crawler)Scooter/1.0 scooter@pa.dec.com scooter.pa-x.dec.com AltaVista
(instant spider)Scooter/1.0 ww2.altavista.digital.com
-or-
add-url.altavista.digital.comEuroseek Arachnoidea *.euroseek.net
such as: infra.euroseek.netExcite
(mega spider)ArchitextSpider crawl*.atext.com
such as: crawl2.atext.comExcite
(fresh spider)ArchitextSpider crimpshrine.atext.com Excite
(NewsTracker Spider)ExciteSpider NewsTracker/1.0 efy-spider.excite.com HotBot Slurp/2.0 *.inktomi.com
such as: j2001.inktomi.com
or j10.inktomi.comInfoseek
(regular crawler)InfoSeek Sidewinder/0.9 *-bbn.infoseek.com
such as: wilbur-bbn.infoseek.com
or
204.162.98.* or 204.162.96.*
such as: 204.162.98.90Lycos
(regular spider)Lycos_Spider_(T-Rex)/3.0 lycosidae.lycos.com
or
spider*.srv.pgh.lycos.com
such as: spider3.srv.pgh.lycos.comLycos
(Add URL spider)Lycos_Spider_(T-Rex)/3.0 www.lycos.com Northern Light Gulliver/1.2 taz.northernlight.com WebCrawler
(regular spider)WebCrawler/3.0_Robot libwww/5.0a
or
WebCrawler/3.0 Robot
libwww/5.0a*.webcrawler.com
such as: tripod.webcrawler.comWebCrawler
(Add URL spider)WebCrawler-AddURL/2.0 *webcrawler.com:
such as: wc-s5.webcrawler.comWebCrawler
(URL Status Checker)WebCrawler-Status/1.0 info.webcrawler.com
|
|
|
...easy-to-use, intelligent, internet robot that builds a link directory and creates link trades for you! |
|
Enroll NOW for our FREE 'Real-World Marketing For Webmasters' Series
This informative series is a must-have for all webmasters.. It's helped thousands of web masters drive millions of visitors to their sites. Don't be left out.
David Notestine's Mentoring Lessons - One-of-a-kind lessons from David Notestine's actual Internet consulting and mentoring work. Each lesson has real-life webmaster problems, questions, answers and suggestions. Everything of importance to webmasters is covered. Each lesson has forum type discussions with David Notestine and/or his consulting clients. Unique to the Internet.
David Notestine's Marketing Messages - Controversial, real-world marketing concepts from the creator of Zeus.
Zeus Robot News and Alerts - Important news and alerts about the Zeus Internet Marketing Robot.
Zeus Power Tips - Quick and easy tips on the proper use of Zeus, hints to increase your link trade response rates, fine-tuning your software and hundreds of undocumented Zeus features.
Web Site created and maintained by Saturn Web Designs
 Top of page |
Privacy statement, copyright and contact information Copyright © 1999 - 2008, Cyber-Robotics Inc, all rights reserved Microsoft, Microsoft Access and Microsoft Office are registered trademarks of Microsoft Corporation |
 |