 |
 |
|
 |
 |
RobotsRobots or “spiders” are programs that are sent out on to the internet from search engines, such as Google, that automatically index websites. The Google spider (known as Googlebot) does a semi-regular "crawl" into the interior pages of sites that are already in its database. This crawl is used by Google to index many or all of the interior pages of websites as well as find new pages (and new sites) to index. It is possible tell whether a website has been crawled by looking for the annotation "Googlebot" in the visitor logs on the web server. After the website has been crawled, new pages are placed in Google's cache for the website. Googlebot visits sites with a high Google Page Rank on the home page (typically PR5 or higher) virtually every day. Lower ranked sites will get visits at longer intervals. |
