Google's new crawler 'Googlebot 2.0' collects more than just textual information

Until now, it was thought that only text information was crawled, but it seems to be a surprising guess that the page is thumbnailed in order to derive search results more accurately.

Googlebot gets smarter; Smile for the picture. | Adsense bits

In short, until now, I was only looking at the page like Lynx, the originator of the text browser, and I mainly collected and analyzed text information roughly, but from around 2005 the behavior of the crawler began to change. Now, it is said that it acquires and analyzes the CSS and JavaScript of the page, recognizes the top and bottom of the page properly, and crawls considering the layout of where and what is placed.

Certainly, looking back at the access log, Googlebot used to take pages to the dark clouds in an unreasonable order, but recently I clicked the most prominent link, and on the next page I clicked each link After that, I went back to the table of contents page and clicked again ... and it behaved like a 'human'.

So, as for the reason why it is necessary to keep the page layout as if taking a snapshot and browse the thumbnail image like this, the above site says 'Optimal delivery of advertisements. Isn't it for the sake of it? '

A further independent guess from the above idea is that the places of ads that humans often see are more frequent from top to bottom, with the highest being just above the middle of the page. So, Google's advertisement distribution is an auction type, so if you pay a high amount of money, it will be displayed preferentially to the top. In other words, the results of advertisements appearing in search results are simply from top to bottom, but in the case of sites such as personal blogs, it is determined whether the tag is embedded in the doco, and it is sold by the optimum auction. Isn't it delivered to the place according to the price? It will be.

Moreover, I think this guess is probably a 'hit'. If you have an ad account, it is best to create your own trial ad display site and actually deliver it. It can have terrifying results.