AdIdxBot

Bingbot is Microsoft’s primary web crawler, responsible for discovering and indexing content for Bing Search and other Microsoft services. The crawler fetches HTML, structured data, images, and metadata to understand page relevance and ranking signals. Crawl activity varies based on site authority, update frequency, and sitemap signals. Its purpose is to keep Bing’s search index fresh, accurate, and aligned with user search intent across Microsoft platforms. RobotSense.io verifies Bingbot using Microsoft’s official validation methods, ensuring only genuine Bingbot traffic is identified.

BingPreview is Microsoft’s rendering and compatibility crawler used to evaluate how webpages appear in browsers and Bing search features. It fetches pages to test layout, mobile responsiveness, JavaScript rendering, and visual elements. These checks help Bing understand how content will display in search results and improve snippet generation and ranking signals tied to user experience. Crawl activity is moderate and often concentrated on pages important to Bing’s index. Its purpose is to simulate real-browser behavior and refine Bing’s presentation quality. RobotSense.io verifies BingPreview using Microsoft’s official validation methods, ensuring only genuine BingPreview traffic is identified.

BingVideoPreview is Microsoft’s crawler for fetching video-related content to generate previews, thumbnails, and metadata for Bing’s video search experiences. It retrieves video files, poster images, structured data, captions, and surrounding context. This crawler does not perform full-site indexing; instead, it focuses specifically on video assets and the information required to power Bing’s video carousels and preview interfaces. Activity is targeted and relatively low-volume, driven by pages that contain or reference video content. RobotSense.io verifies BingVideoPreview using Microsoft’s official validation methods, ensuring only genuine BingVideoPreview traffic is identified.

MicrosoftPreview is a Microsoft crawler used to render webpages in a browser-like environment for testing, feature evaluation, and content understanding across Microsoft services. It performs fetches that simulate modern browser behavior, including JavaScript execution, layout rendering, and metadata extraction. Unlike Bingbot, it is not used for core indexing but for assessing how pages display in Microsoft products. Activity is moderate and focused on pages relevant to rendering quality checks. Its purpose is to enhance visual accuracy and user experience across Microsoft’s platforms. RobotSense.io verifies MicrosoftPreview using Microsoft’s official validation methods, ensuring only genuine MicrosoftPreview traffic is identified.

by Google

AdsBot-Google is Google’s crawler responsible for evaluating landing pages used in Google Ads campaigns. It performs desktop-focused checks on page quality, load speed, relevance, and policy compliance. These assessments directly influence ad quality scores, cost efficiency, and overall eligibility. Blocking AdsBot prevents Google from reviewing landing pages, which can degrade or disable ad performance. Crawl activity is selective and tied to active or recently modified ad campaigns rather than broad indexing. Its purpose is to ensure that advertisers maintain fast, trustworthy, and policy-compliant landing pages. It ignores the global user agent (*) rule. RobotSense.io verifies AdsBot/AdsBot-Google using Google’s official validation methods, ensuring only genuine AdsBot/AdsBot-Google traffic is identified.

by Google

[This crawler is officially retired as per Google] AdsBot-Google-Mobile bot is Google’s mobile-focused crawler used to evaluate the landing page experience for Google Ads. It simulates mobile device conditions to assess page quality, load performance, mobile usability, and policy compliance. These evaluations directly influence Google Ads quality scores and ad eligibility. If you run ads, blocking it may negatively affect ad performance because Google cannot verify the mobile landing page experience. Crawl activity is targeted and low-volume, triggered when ads are created, updated, or actively running. Its purpose is ensuring advertisers provide fast, compliant, and user-friendly mobile pages. It ignores the global user agent (*) rule. RobotSense.io verifies AdsBot Mobile Web/AdsBot-Google-Mobile using Google’s official validation methods, ensuring only genuine AdsBot Mobile Web/AdsBot-Google-Mobile traffic is identified.

by Google

Mediapartners-Google is Google’s crawler dedicated to evaluating webpages for Google AdSense. It scans pages to understand content, layout, and context so Google can deliver relevant ads and optimize revenue for publishers. Unlike Googlebot, this crawler does not index content for Search - its role is purely advertising-related. Blocking it may prevent AdSense from analyzing pages and serving targeted ads effectively. Crawl activity is generally light and focused on pages where AdSense code is present, helping Google match ad inventory with page themes and user interests. It ignores the global user agent (*) rule. RobotSense.io verifies AdSense / Mediapartners-Google using Google’s official validation methods, ensuring only genuine AdSense / Mediapartners-Google traffic is identified.

by Meta / Facebook

Meta-ExternalAds is Meta’s crawler used to evaluate landing pages associated with ads running on Facebook, Instagram, and other Meta platforms. It performs targeted checks to assess page load behavior, policy compliance, redirects, content quality, and overall ad safety. These fetches are ad-driven, not general web crawling, and help Meta determine whether landing pages meet advertising standards. Blocking it may affect ad review accuracy or eligibility. Crawl activity is focused, low-volume, and typically triggered when advertisers submit new ads, update creatives, or undergo automated policy reviews. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalAds using Meta’s official validation methods, ensuring only genuine Meta-ExternalAds traffic is identified.

by Google

[This crawler is officially retired as per Google] AdsBot-Google-Mobile-Apps is Google’s crawler designed to evaluate landing pages used in mobile app promotion campaigns on Google Ads. It simulates a mobile app–driven environment to assess page load speed, relevance, usability, and compliance with Google Ads policies. These evaluations directly influence ad quality scores and campaign performance. Blocking it may prevent Google from validating landing pages for app ads, potentially lowering ad effectiveness or eligibility. Crawl activity is targeted and on-demand, typically occurring when app promotion ads are created, updated, or actively running. It ignores the global user agent (*) rule. RobotSense.io verifies Mobile Apps Android / AdsBot-Google-Mobile-Apps using Google’s official validation methods, ensuring only genuine Mobile Apps Android / AdsBot-Google-Mobile-Apps traffic is identified.