A while back I wrote an article pointing out one of the features available to webmasters at Google Webmaster Central – ability to set a custom crawl rate. At the time I thought it was an outstanding idea. It is still is a good solution should googlebot visit your site too often causing issues with bandwidth (even though I have never heard of such scenario).
Webmasters however should exercise caution when increasing the crawl rate via Google Webmaster Central. A few weeks ago I was involved in an upgrade of a website with 2000+ pages. One of the side effects of the upgrade was that we ended up with 2000+ new urls. Unfortunately there was no easy way to implement 2000+ redirects on the website. But, since most of the traffic came from PPC campaigns, we decided to let Google re-index the website on it’s own. Exception was of course that we used Google Webmaster Central to implement request for a more frequent crawl by Google. We increased the crawl rate by some 3000% only to receive the following notice a few days later:
Dear Webmaster,
Google has algorithms that determine how much to crawl each site. Our goal is to crawl as many page from your site, http://www.xxxxx.com/, as we can without overwhelming your server’s bandwidth.
For your site, http://www.xxxxx.com/, you have set a very low crawl rate which is preventing us from accessing your great content. We recommend you set the crawl rate option to “Let Google determine my default”, or, if you prefer to maintain a custom setting, increasing the rate.
Thanks,
The Google Web Crawling Team
In fact the Google had not indexed a single URL from the new sitemap. Not even the home page (which is/was the strongest page of the website) was re-indexed. So we followed Google’s instruction and within 24 hours our new pages began to appear in the index.
So if you are thinking of increasing the crawl rate in Google Webmaster Central – exercise caution or you may run the risk of not increasing the the crawl rate enough for Google to do it’s job.
Great site, I think we can all learn something from this.
Never new theres an option on how you’ll let google crawl your site, better check this feature.
I also tried this before and Google also emailed me, I guess Google doesn’t want manipulating the crawl rate of our site.
We usually leave the Google defaults. We have had great results in Google’s search results.
Getting indexed is pretty easy. Just link to your new site from a few established ones and your site should get indexed very quickly. Getting your site to rank well is a different story.