 |
| Google Crawlers |
Googlebot and Adsbot-Google are different crawlers for different purposes. Actually, these are the program that browse the web in an automated manner by following one web-page to another. Without the web crawlers , there would be no other way to tell search engines that your website has new and fresh content. Google's main crawler is called Google bot ( For Organic results).
List of Google Crawlers
Besides Google’s web search crawler, they actually have 9 additional web crawlers:| Web Crawler | User-Agent | |
|---|---|---|
| Web Crawler | User-Agent | |
| Googlebot News | Googlebot-News | |
| Googlebot Images | Googlebot-Image/1.0 | |
| Googlebot Video | Googlebot-Video/1.0 | |
| Google AdsBot | AdsBot-Google | |
| Google app crawler | AdsBot-Google-Mobile-Apps | |
| Google Adsense | Mediapartners-Google | |
| Google Mobile Adsense | (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html) | |
| Google Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| Google Mobile (featured phone) | SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
By placing a robots.txt file at the root of your web server you can define rules for web crawlers such as allow or disallow that they must follow.
It is highly recommended to allow Google bots to crawl your website but some times it may require.
When to Prevent Access of Google Bot
I have faced this issue when i was running a client adwords campaign in medical niche and Google was disapproving our all ads just for using a single restricted word in our website. (somewhere in website directory but not in our landing page).How to block access of Google Adwords Bots
If you really want to block access of Google Adwords crawler from accessing your other folders and pages in your sites, then you need to make some changes on robots.txt file in your directory.1. First, you need to access website root directory
2. Open your robots.txt file in notepad.
3. Put below mentioned code in the file.
User-agent: AdsBot-Google
Disallow: /Your-Page/
From this way, You can prevent access of Google Adwords bot. This will not impact your google organic results. Hope this article help !!
