MicrosoftPreview

AdIdxBot is Microsoft’s advertising-focused crawler used to evaluate landing pages associated with Microsoft Advertising (formerly Bing Ads). It performs targeted fetches to assess page load speed, relevance, redirects, content quality, and policy compliance. These checks help determine ad quality scores, eligibility, and overall user experience. Blocking AdIdxBot, may limit Microsoft’s ability to review landing pages, potentially affecting ad performance. Crawl activity is selective and low-volume, typically triggered when advertisers create or modify campaigns. Its purpose is to ensure that ad destinations meet Microsoft’s standards for safety, usability, and relevance. RobotSense.io verifies AdIdxBot using Microsoft’s official validation methods, ensuring only genuine AdIdxBot traffic is identified.

Bingbot is Microsoft’s primary web crawler, responsible for discovering and indexing content for Bing Search and other Microsoft services. The crawler fetches HTML, structured data, images, and metadata to understand page relevance and ranking signals. Crawl activity varies based on site authority, update frequency, and sitemap signals. Its purpose is to keep Bing’s search index fresh, accurate, and aligned with user search intent across Microsoft platforms. RobotSense.io verifies Bingbot using Microsoft’s official validation methods, ensuring only genuine Bingbot traffic is identified.

BingPreview is Microsoft’s rendering and compatibility crawler used to evaluate how webpages appear in browsers and Bing search features. It fetches pages to test layout, mobile responsiveness, JavaScript rendering, and visual elements. These checks help Bing understand how content will display in search results and improve snippet generation and ranking signals tied to user experience. Crawl activity is moderate and often concentrated on pages important to Bing’s index. Its purpose is to simulate real-browser behavior and refine Bing’s presentation quality. RobotSense.io verifies BingPreview using Microsoft’s official validation methods, ensuring only genuine BingPreview traffic is identified.

BingVideoPreview is Microsoft’s crawler for fetching video-related content to generate previews, thumbnails, and metadata for Bing’s video search experiences. It retrieves video files, poster images, structured data, captions, and surrounding context. This crawler does not perform full-site indexing; instead, it focuses specifically on video assets and the information required to power Bing’s video carousels and preview interfaces. Activity is targeted and relatively low-volume, driven by pages that contain or reference video content. RobotSense.io verifies BingVideoPreview using Microsoft’s official validation methods, ensuring only genuine BingVideoPreview traffic is identified.

by Amazon

[Amazon Bots can take upto 30 days to read your Robots.txt updates.] Amazonbot is Amazon’s official web crawler, used to discover and fetch webpage content for applications such as Alexa, product-related features, and Amazon’s AI and search systems. Crawl activity varies based on Amazon services that rely on external web content, but it is generally moderate and focused on structured data, text content, and page metadata. Its purpose is to enhance Amazon’s search, AI models, and user-facing features. It ignores the global user agent (*) rule. RobotSense.io verifies Amazonbot using Amazon’s official validation methods, ensuring only genuine Amazonbot traffic is identified.

by Amazon

[Amazon Bots can take upto 30 days to read your Robots.txt updates.] Amzn-User is a bot associated with Amazon services that fetch webpage content on behalf of end users or Amazon applications rather than acting as a general-purpose crawler. It typically appears when Amazon apps, devices, or internal systems request metadata, previews, or content needed for features like link expansion, in-app browsing, or contextual analysis. The traffic is user-driven, not designed for large-scale indexing or scraping. Amzn-User usually performs lightweight, targeted fetches limited to specific URLs users interact with. Its purpose is to support Amazon product experiences by retrieving just enough page data to power user-facing functionality. It ignores the global user agent (*) rule. RobotSense.io verifies Amzn-User using Amazon’s official validation methods, ensuring only genuine Amzn-User traffic is identified.

by Google

[This crawler is officially retired as per Google] DuplexWeb-Google is a Google crawler associated with Duplex and Assistant-related technologies that fetch web content to help generate conversational responses and perform task-oriented actions. It retrieves page information needed to understand structured data, business details, menus, appointment flows, and other interactive elements. Crawl activity is selective and generally tied to user-initiated tasks or systems that prepare content for automated assistance. Its purpose is to support natural-language interactions by ensuring Google’s assistant technologies can interpret and use real-time webpage information accurately. It ignores the global user agent (*) rule. RobotSense.io verifies DuplexWeb-Google using Google’s official validation methods, ensuring only genuine DuplexWeb-Google traffic is identified.

by Meta / Facebook

FacebookExternalHit is Facebook’s (Meta’s) crawler used to fetch webpage content for link previews across Facebook, Messenger, Instagram, and other Meta surfaces. It retrieves metadata such as Open Graph tags, titles, descriptions, images, and structured data. These requests are user-triggered, occurring when someone shares or pastes a URL on a Meta platform. The bot does not index or rank websites and has no connection to search algorithms. Blocking it may prevent accurate link previews. Crawl activity is lightweight and focused on fetching just enough content to generate rich social previews. It ignores the global user agent (*) rule. RobotSense.io verifies FacebookExternalHit using Meta’s official validation methods, ensuring only genuine FacebookExternalHit traffic is identified.

by Google

Google Messages uses a fetcher to retrieve webpage data for generating link previews when users send URLs in chat conversations. It fetches metadata such as titles, descriptions, images, and structured tags (e.g., Open Graph) to render rich previews inside messages. This traffic is strictly user-triggered, occurring only when a link is shared. It is not a crawler for indexing or discovery and has no impact on Google Search rankings. Blocking it may prevent previews from displaying correctly. Activity is lightweight and targeted, focused solely on enhancing the messaging experience with accurate link previews. Google Messages bot does not respect robots.txt rules. RobotSense.io verifies Google Messages using Google’s official validation methods, ensuring only genuine Google Messages traffic is identified.

by Google

GoogleOther is a general-purpose crawler used by Google for internal research, large-scale data analysis, and non–Search-related fetching. It is part of Google’s secondary crawling infrastructure, designed to offload tasks that don’t require the full capabilities or strict policies of Googlebot. GoogleOther typically performs broad but lower-priority fetches, such as machine learning dataset generation or internal experiments. Its activity is generally lightweight compared to Googlebot and is separate from indexing operations that directly influence Google Search results. RobotSense.io verifies GoogleOther using Google’s official validation methods, ensuring only genuine GoogleOther traffic is identified.