Would you imagine that a toy, which is a specialty in Japan, is related to an insect that has bitten Peter Parker. Well, Domo Arigato Mr. Roboto, that is the case in the world wide web.

If you’ve spent any time on the Internet, you may have heard a little about spiders, crawlers, and
robots. These little creatures are programs that literally crawl around the Web, cataloging data so that it can be searched. In the most basic sense all three programs — crawlers, spiders, and robots — are essentially the same. They all “collect” information about each and every web URL.

This information is then cataloged according to the URL on which they’re located and are stored in
a database. Then, when a user uses a search engine to locate something on the Web, the references in the database are searched and the search results are returned.

This entry was posted on Friday, September 5th, 2008 at 9:00 pm and is filed under SEO / SEM.