Have you ever wondered how many types of Google’s websites crawler are there and what’s their use? Here i have tried to cover information about the widely and well know Googlebots and their functionality.
Googlebot
Bot Name : Googlebot
User-agents :- Googlebot
Use :- This well known web crawling bot is used for discovering new and updated pages to Google index. Googlebot uses a process to determine which and how many pages to crawl. Within Googlebot there are two types of bots named Deepbot and Freshbot. As their name suggest Deepbot crawls your website once a month and Freshbot crawls your website on a continuous basis and tries to find content which is updated frequently.
Previously there was a separate bot for crawling News websites but back in August 2011 Google announced that they will be no longer crawling news sites with Googlebot-News bot and from now on the same Googlebot will crawl the news sites too.
Googlebot-Image
Bot Name:- Googlebot Images
User-agents:- Googlebot-Image
Use:- It’s because of this bot you see images when you search for the same. This bot is responsible for crawling the images on web and to place them on Google’s Image Search.
If you want to block google from crawling images of your website, you should inlcude this in your robots.txt in this way, this will block google from crawling any image of your website.
User-agents:- Googlebot-Image
Disallow: /
Googlebot-Video
Bot Name:- Googlebot Video
User-agents:- Googlebot-Video
Use:- This bot is used for discovering video content of your website or on web.
Googlebot-Mobile
Bot Name:- Googlebot Mobile
User-agents:- Googlebot-Mobile
Use :- Just like Googlebot is used for crawling web search result Googlebot-Mobile is used for crawling mobile web search result.
AdsBot-Google
Bot Name:- Google Ads bot
User-agents:- AdsBot-Google
Use:- This bot is used to monitor AdWords landing page quality. In case you are running PPC campaign for your product/website make sure that you haven’t accidentally blocked the AdsBot from crawling your landing page. Google uses the bid amount which you gives to dertermine your ad posistion . This bot will only visit your landing page which you have used in your adword campaign.
Was this post helpful? If yes, please feel free to share it.
1 Comment