Googlebot is your trusted friend

If you’re into tracking your visitors, one visitor that is always there, and very very often, is Googlebot. You may have thought about it as an evil or a good. But what is it ?

Googlebot will crawl every page on your site, unless you really mess up or spam the robots. But you’ll still need the robots.txt file. If you don’t have it, make one. Reduce your errors and Googlebot will be must friendlier.

But Googlebut makes your tracking harder. Using webtrends, you’ll probably like to sort away bots, which should be an easy task in the newer versions of Webtrends. Most advanced tracking-software or services have this ability.

Googlebot and google will rate your pages, index them and present them according to the rating. So, go do some deaper research on Googlebot, and your site will be easier to find out there. That is, if you want that.

You should start at Googlebots “homepage” http://www.google.com/webmasters/bot.html, there you should get your first insight in this bot.

Then there are the other bots. MSN, Yahoo etc etc. Take a look in your logfiles or your online trackingtool, and follow the links that the bots leave behind.