Telling Google How Often To Crawl Your Website
If you are using Google Webmaster Tools you are probably aware that, among many things that these tools offer, there is the ability to submit your web site’s sitemap to help Google discover pages on your website. Google Webmasters Tools can also provide some insightful information about your website. Among many different statistics you will see the page containing information about about “crawl statistics”. You will find there a graphic of Googlebot’s activity on your website over the period of last 90 days. Just like the one you see below.

In the past Google did not offer webmasters many options or saying about how ofter they can or will crawl your website. While from time to time, after determining that your website may use a faster crawl rate, they would offer you an option to change this rate. However over the period last few months I was never offered that option in Google Webmaster Tools for my websites. So I have trusted Google, as most of us do, to crawl and discover the websites in their own time, and have completely forgotten about the “crawl rate” settings.
It appears, however, that recently, Google Webmaster Tools have opened the option to set a custom crawl rate for the websites you have listed in your account:

The above option can be found under the main settings in your Google Webmaster Tools account, and it is quiet self-explanatory. Stetting a “slower crawl rate” will reduce Googlebot’s visits to your website. I would recommend not to set a rate slower of what Google is already doing, unless your are experiencing issues with bandwidth. The slower crawl rate will probably result in delay with indexing your latest content. If anything, you may want to experiment with a “faster rate”, if you think it takes too long for your latest content to appear in Google’s index.
For time being I will not customize the crawl rate for this website, as I am pleased with what Google is doing now. I will however experiment with few other websites with intention to report here how can you benefit for this option, if you can benefit at all.
For example today I have noticed that there are 57 pages on this blog that have “duplicate” meta descriptions. While I can not be certain how much effect this actually has on how well my blog ranks in the search results. I am assuming there may be some sort of penalty associated with those “dupes”.
It is very easy to find out if Google has issues with duplicate meta tags on your website. All you need is to log into your account and click on “Diagnostic” menu and then on “Content Analysis”. If Google has nothing to report under that menu I guess you can consider it a good news. In may case as shown in one of the screen shots above I have an issue with at leas 57 pages containing meta tags that similar or straight duplicate from other pages. If you are running a WordPress blog and an plugin that automates the process of creating metatags you may run into a similar problem. What is the solution? First you can probably block Google bot via robots.txt from indexing the duplicates, but I would not recommend doing so. Chances are those duplicate are result of category or archive pages on your website.

