What is a Web Crawler or Spider or Bot ???

A crawler is a program that regularly visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web will have such program, which is also known as a "spider" or a "bot". Crawlers are typically programmed to visit sites that have been submitted by their owners as new or updated. Entire sites or specific pages can be selectively visited and indexed. Crawlers apparently gained the name because they crawl through a site a page at a time, following the links to other pages on the site until all pages have been read. In another terms we can describe a web crawler as a blind child with a clear sense of right and wrong. We call it blind because it doesn't see all the fancy graphics and creative flash and it can only see the text and alternative text on your site. It is termed as a child because it has to be guided around your website by clear paths that it can understand. The effective sense of right and wrong comes from it having the ability to determine if something is not right. i.e., if it comes across some red text that is on top of a red background it will see it as hidden text or if it sees a page redirect to a different site without any reason for it, it will see it as a gateway page, both of which should be avoided.