Read This Usefull Information about Spiders, Crawlers and Bots by SEO Expert D Hari Babu ( Seo Trainer and Consultant from Hyderabad- www.srihithatechnologies.com)
Spider:
" A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot." Spiders are typically programmed to visit sites that have been submitted by their owners as new or updated." "Entire sites or specific pages can be selectively visited and indexed. Spiders are called spiders because they usually visit many sites in parallel at the same time, their "legs" spanning a large area of the "web." Spiders can crawl through a site's pages in several ways. One way is to follow all the hypertext links in each page until all the pages have been read." In order to allow spiders to effectively crawl your site, it helps to have easy navigation that will allow all pages to be linked to all other pages through some simple menu bar - then the number of pages you have "indexed" can be high and this wil help the over all rankingof your site.
Googlebot
Google Bot is Google's web-crawling robot. Google freely describes basic information about this program and what it does at www.google.com/bot.html
In case if you want spider to not follow your website or you dont want spider to read your website then read this information.
There may be some pages on a website that you do not want to show up in Google, for example pages that are password protected or other content with difficult to find links. To keep the Googlebot from following links on your pages to other pages or documents, you should add a meta tag into the header tags of your page. The code looks like this:
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment