Understand search engine crawlers

 
 Understand search engine crawlers

Did you ever wonder how all those pages got into the search engines in the first
place? There's a magic search engine genie that flies from server to server waving
a magic wand; not really but close. Actually, there is a computer program called a
crawler (or sometimes a spider or robot or 'bot) that lives on the search engine's
servers. Its job is to surf the Web and save anything it finds. It starts by visiting sites
that it already knows about and after that, follows any links that it finds along the
way. At each site that it visits, it grabs all the HTML code from every single page it
can find and saves that on its own servers.
Later, an indexing server will take that HTML code, examine it, parse it, filter it,
analyze it, and some other secret stuff (a lot like waving that magic wand). Finally,
your site is saved into the search engine's index. Now, it is finally ready to be served
up as a search result. Total time elapsed? About two minutes.
One important thing to note here is that search engine crawlers follow the same links
that you do. That means that if you can't click the link, then there's a good chance
that the crawler can't click the link either. Fortunately Google does a great job of
following JavaScript links. That's why those tools are sometimes
called spider food.