Crawling web pages so that they can be added to a search engine's results page.
For example, if you were to type in 'Tech savvy internet sorts fmttm' 2 seconds after you posted this thread, it would not appear in a Google search.
But, once it has been crawled by these 'robots', they will...