my web host warned me due to high CPU usage.
my Site been crawled by googlebot – about 28000 hits and msnboot about 1000 hits. Is this possible that indexing caused a high cpu usage? I been advised to setup a robots.txt to limit googlebot hits, but I am not sure how to do this.
my Site been crawled by googlebot – about 28000 hits and msnboot about 1000 hits. Is this possible that indexing caused a high cpu usage? I been advised to setup a robots.txt to limit googlebot hits, but I am not sure how to do this.
I think there is a lot of traffic on your site and typically Google is pretty light weight about this stuff
.robots.txt won’t help you here. You can sign up for a free account at Google Webmaster Tools and in there you can ask them to crawl your site less often. But robots.txt is all or nothing – you can say don’t ever crawl these set of pages, but that’s NOT what you want!
Googlebot isn’t causing it. Most likely a script he is using has a memory leak or something to a similiar extent.
What scripts u r using,custom-written or commercial/open source?
ask your web host, which scripts are responsible for high cpu load?
My web host told me only about these hits.
it’s doubtful that Googlebot is causing this…it never affects CPU%.
Ananova 