Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site. Google's crawlers are also programmed such that they try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500 errors mean "slow down").

Here’s an example. This is the results page for “fish oil.” In this case, the top results are about health benefits. That means users are likely looking for these types of articles. Rather than the best fish oil products, for example.