Telling Google How Often To Crawl Your Site- Not A Good Idea After All.

Vlad Zablotskyy — Wed, 23 Sep 2009 15:58:54 +0000

A while back I wrote an article pointing out one of the features available to webmasters at Google Webmaster Central – ability to set a custom crawl rate. At the time I thought it was an outstanding idea. It is still is a good solution should googlebot visit your site too often causing issues with bandwidth (even though I have never heard of such scenario).

Webmasters however should exercise caution when increasing the crawl rate via Google Webmaster Central. A few weeks ago I was involved in an upgrade of a website with 2000+ pages. One of the side effects of the upgrade was that we ended up with 2000+ new urls. Unfortunately there was no easy way to implement 2000+ redirects on the website. But, since most of the traffic came from PPC campaigns, we decided to let Google re-index the website on it’s own. Exception was of course that we used Google Webmaster Central to implement request for a more frequent crawl by Google. We increased the crawl rate by some 3000% only to receive the following notice a few days later:

Dear Webmaster,
Google has algorithms that determine how much to crawl each site. Our goal is to crawl as many page from your site, http://www.xxxxx.com/, as we can without overwhelming your server’s bandwidth.
For your site, http://www.xxxxx.com/, you have set a very low crawl rate which is preventing us from accessing your great content. We recommend you set the crawl rate option to “Let Google determine my default”, or, if you prefer to maintain a custom setting, increasing the rate.
Thanks,
The Google Web Crawling Team

In fact the Google had not indexed a single URL from the new sitemap. Not even the home page (which is/was the strongest page of the website) was re-indexed. So we followed Google’s instruction and within 24 hours our new pages began to appear in the index.

So if you are thinking of increasing the crawl rate in Google Webmaster Central – exercise caution or you may run the risk of not increasing the the crawl rate enough for Google to do it’s job. Similar Posts:

Telling Google How Often To Crawl Your Website

Vlad Zablotskyy — Fri, 26 Dec 2008 05:39:33 +0000

If you are using Google Webmaster Tools you are probably aware that, among many things that these tools offer, there is the ability to submit your web site’s sitemap to help Google discover pages on your website. Google Webmasters Tools can also provide some insightful information about your website. Among many different statistics you will see the page containing information about about “crawl statistics”. You will find there a graphic of Googlebot’s activity on your website over the period of last 90 days. Just like the one you see below.

In the past Google did not offer webmasters many options or saying about how ofter they can or will crawl your website. While from time to time, after determining that your website may use a faster crawl rate, they would offer you an option to change this rate. However over the period last few months I was never offered that option in Google Webmaster Tools for my websites. So I have trusted Google, as most of us do, to crawl and discover the websites in their own time, and have completely forgotten about the “crawl rate” settings.

It appears, however, that recently, Google Webmaster Tools have opened the option to set a custom crawl rate for the websites you have listed in your account:

The above option can be found under the main settings in your Google Webmaster Tools account, and it is quiet self-explanatory. Stetting a “slower crawl rate” will reduce Googlebot’s visits to your website. I would recommend not to set a rate slower of what Google is already doing, unless your are experiencing issues with bandwidth. The slower crawl rate will probably result in delay with indexing your latest content. If anything, you may want to experiment with a “faster rate”, if you think it takes too long for your latest content to appear in Google’s index.

For time being I will not customize the crawl rate for this website, as I am pleased with what Google is doing now. I will however experiment with few other websites with intention to report here how can you benefit for this option, if you can benefit at all.Similar Posts:

Go Beyond MLS » Google Webmster Tools

Telling Google How Often To Crawl Your Site- Not A Good Idea After All.

Telling Google How Often To Crawl Your Website