Official Google Webmaster Central Blog: All About Googlebot

Webmaster Central Blog

Official news on crawling and indexing sites for the Google index

All About Googlebot

Saturday, August 19, 2006

the Bot Obedience courseWebmasterWorld forumsIf my site is down for maintenance, how can I tell Googlebot to come back later rather than to index the "down for maintenance" page?statusWhat should I do if Googlebot is crawling my site too much?contact usIs it better to use the meta robots tag or a robots.txt file?If my robots.txt file contains a directive for all bots as well as a specific directive for Googlebot, how does Googlebot interpret the line addressed to all bots?User-agent: *
Disallow: /

User-agent: Googlebot
Disallow: /cgi-bin/User-agent: *
Disallow: /robots.txt analysis toolwebmaster help center

Labels: crawling and indexing

Hey! Check here if your site is mobile-friendly.

Labels

Archive

2020
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2019
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2018
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2017
- Dec
- Nov
- Oct
- Sept
- Aug
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2016
- Dec
- Nov
- Oct
- Sept
- Aug
- Jun
- May
- Apr
- Mar
- Jan

2015
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- May
- Apr
- Mar
- Feb
- Jan

2014
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2013
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2012
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2011
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2010
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2009
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2008
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2007
- Dec
- Nov
- Oct
- Sept
- Aug
- Jul
- Jun
- May
- Apr
- Mar
- Feb
- Jan

2006
- Dec
- Nov
- Oct
- Sept
- Aug

Feed

Give us feedback in our Product Forums.

Subscribe via email

Enter your email address:

Delivered by FeedBurner

Google
Privacy
Terms