Google

APIs-Google is a service crawler used by Google to verify and interact with endpoints tied to various Google APIs. It is typically triggered when applications, scripts, or integrations using Google services need to fetch or validate external resources. Common use cases include OAuth flows, link previews, data validation, push notification messages, and API-driven checks performed on behalf of Google products. Crawl activity is usually low-volume and event-driven, reflecting specific API operations rather than broad crawling or indexing associated with Google Search. It ignores the global user agent (*) rule. RobotSense.io verifies APIs-Google using Google’s official validation methods, ensuring only genuine APIs-Google traffic is identified.

Mediapartners-Google is Google’s crawler dedicated to evaluating webpages for Google AdSense. It scans pages to understand content, layout, and context so Google can deliver relevant ads and optimize revenue for publishers. Unlike Googlebot, this crawler does not index content for Search - its role is purely advertising-related. Blocking it may prevent AdSense from analyzing pages and serving targeted ads effectively. Crawl activity is generally light and focused on pages where AdSense code is present, helping Google match ad inventory with page themes and user interests. It ignores the global user agent (*) rule. RobotSense.io verifies AdSense / Mediapartners-Google using Google’s official validation methods, ensuring only genuine AdSense / Mediapartners-Google traffic is identified.

AdsBot-Google is Google’s crawler responsible for evaluating landing pages used in Google Ads campaigns. It performs desktop-focused checks on page quality, load speed, relevance, and policy compliance. These assessments directly influence ad quality scores, cost efficiency, and overall eligibility. Blocking AdsBot prevents Google from reviewing landing pages, which can degrade or disable ad performance. Crawl activity is selective and tied to active or recently modified ad campaigns rather than broad indexing. Its purpose is to ensure that advertisers maintain fast, trustworthy, and policy-compliant landing pages. It ignores the global user agent (*) rule. RobotSense.io verifies AdsBot/AdsBot-Google using Google’s official validation methods, ensuring only genuine AdsBot/AdsBot-Google traffic is identified.

AdsBot-Google-Mobile bot is Google’s mobile-focused crawler used to evaluate the landing page experience for Google Ads. It simulates mobile device conditions to assess page quality, load performance, mobile usability, and policy compliance. These evaluations directly influence Google Ads quality scores and ad eligibility. If you run ads, blocking it may negatively affect ad performance because Google cannot verify the mobile landing page experience. Crawl activity is targeted and low-volume, triggered when ads are created, updated, or actively running. Its purpose is ensuring advertisers provide fast, compliant, and user-friendly mobile pages. It ignores the global user agent (*) rule. RobotSense.io verifies AdsBot Mobile Web/AdsBot-Google-Mobile using Google’s official validation methods, ensuring only genuine AdsBot Mobile Web/AdsBot-Google-Mobile traffic is identified.

Chrome Web Store fetcher or Google-CWS is a Google user-agent associated with Chrome Web Services, typically used for link preview generation, safe browsing checks, and content fetching triggered by Chrome features. It performs lightweight requests to retrieve metadata, page titles, favicons, and safety signals from URLs that developers provide in the metadata of their Chrome extensions and themes. This bot is not a search crawler and does not influence Google Search indexing or rankings. Activity occurs when Chrome or Google services need to quickly inspect a URL for previews, safety evaluation, or rendering behavior. It ignores the global user agent (*) rule. RobotSense.io verifies Chrome Web Store fetcher using Google’s official validation methods, ensuring only genuine Chrome Web Store fetcher traffic is identified.

[This crawler is officially retired as per Google] DuplexWeb-Google is a Google crawler associated with Duplex and Assistant-related technologies that fetch web content to help generate conversational responses and perform task-oriented actions. It retrieves page information needed to understand structured data, business details, menus, appointment flows, and other interactive elements. Crawl activity is selective and generally tied to user-initiated tasks or systems that prepare content for automated assistance. Its purpose is to support natural-language interactions by ensuring Google’s assistant technologies can interpret and use real-time webpage information accurately. It ignores the global user agent (*) rule. RobotSense.io verifies DuplexWeb-Google using Google’s official validation methods, ensuring only genuine DuplexWeb-Google traffic is identified.

Feedfetcher is Google’s crawler responsible for retrieving RSS and Atom feeds used in Google News, Google Reader (historically), and other syndication-based services. It fetches feed URLs rather than full webpages. The bot does not index content for Google Search and does not follow links within feeds; its role is purely to collect updates for subscribed users or Google systems that aggregate feed content. Most publishers allow it to ensure timely distribution of updates. Crawl activity is periodic and lightweight, triggered when feed subscribers or internal services request refreshes. It ignores the global user agent (*) rule. RobotSense.io verifies Feedfetcher using Google’s official validation methods, ensuring only genuine Feedfetcher traffic is identified.

[This crawler is officially retired as per Google] Google Favicon is a specialized Google crawler that retrieves website favicons for use across Google Search, Chrome, and other Google products. It fetches small icon files such as favicon.ico or declared alternative icons in HTML. This bot does not index page content or affect Search rankings; its role is purely to collect icons that visually represent sites in SERPs and browser surfaces. Most sites allow it since its requests are lightweight. Crawl activity is minimal and typically occurs when Google detects new or updated favicon assets.

Google NotebookLM is an AI-assisted research tool from Google that can fetch page content when users provide URLs as sources. Its fetcher performs targeted requests to retrieve text, metadata, and structural information needed for summarization, analysis, and note-building. This activity is user-initiated, not a general-purpose crawler, and it does not influence Google Search indexing. Crawl volume is minimal and tied directly to user actions within NotebookLM. Its purpose is to supply accurate page content to support AI-driven research and knowledge organization. It ignores the global user agent (*) rule. RobotSense.io verifies Google NotebookLM using Google’s official validation methods, ensuring only genuine Google NotebookLM traffic is identified.

Google Pinpoint is a research-focused tool used by journalists and investigative teams to analyze large volumes of documents. When users add URLs as sources, Pinpoint’s fetcher retrieves page content to extract text, metadata, and searchable information. The tool performs strictly user-initiated fetches, not broad crawling. Activity is low-volume and targeted to the exact URLs a user imports. Pinpoint does not influence Google Search indexing; its purpose is solely to help researchers organize, search, and analyze content within their private Pinpoint collections. It ignores the global user agent (*) rule. RobotSense.io verifies Google Pinpoint using Google’s official validation methods, ensuring only genuine Google Pinpoint traffic is identified.

Google Publisher Center is a platform that allows news publishers to manage how their content appears across Google News surfaces. When publishers submit feeds, sections, or site updates, Google may fetch associated URLs using Publisher Center–related user-agents to verify content, metadata, and feed accuracy. These fetches are not broad crawls; they are targeted checks tied to publisher actions such as updating feeds, article structures, or publication settings. Blocking it can disrupt feed validation or delay updates in Google News. Activity is typically light, triggered by publisher configuration changes or system refresh cycles. It ignores the global user agent (*) rule. RobotSense.io verifies Google Publisher Center / GoogleProducer using Google’s official validation methods, ensuring only genuine Google Publisher Center / GoogleProducer traffic is identified.

Google Read Aloud is a Google service that fetches webpage content so it can be converted into spoken audio for users. It retrieves the main text, metadata, and structural elements needed to generate accurate audio playback. These requests are user-initiated, not part of Google Search crawling, and do not affect indexing or rankings. Crawl activity is minimal and focused solely on the specific pages users choose to listen to. Its purpose is to enhance accessibility and on-the-go reading experiences. It ignores the global user agent (*) rule. RobotSense.io verifies Google Read Aloud using Google’s official validation methods, ensuring only genuine Google Read Aloud traffic is identified.

Google Site Verifier is a Google service that confirms ownership of websites for tools like Google Search Console, Google Workspace, and other property-based products. It performs targeted requests to verify the presence of verification tokens, such as HTML files, meta tags, or DNS records. These checks are one-time or periodic, triggered when a user initiates verification or when Google revalidates ownership. The bot does not crawl or index content and has no impact on Search rankings. It's activity is minimal, focused solely on confirming site ownership credentials. It ignores the global user agent (*) rule. RobotSense.io verifies Google Site Verifier / Google-Site-Verification using Google’s official validation methods, ensuring only genuine Google Site Verifier / Google-Site-Verification traffic is identified.

Google StoreBot is Google’s crawler responsible for fetching and validating data related to product listings, merchant feeds, and eCommerce pages used across Google Shopping surfaces. It helps Google evaluate product availability, pricing, structured data, and landing page quality. StoreBot works alongside Merchant Center systems to ensure product information is accurate, up-to-date, and compliant with Google’s listing requirements. Crawl behavior is focused, lightweight, and typically triggered by updates to product feeds or changes detected on merchant landing pages. RobotSense.io verifies Google StoreBot using Google’s official validation methods, ensuring only genuine Google StoreBot traffic is identified.

Google-CloudVertexBot is a crawler associated with Google Cloud’s Vertex AI ecosystem, used to retrieve web content for AI model evaluation, dataset preparation, and automated tool workflows within Vertex pipelines. The bot’s crawling is typically task-specific-triggered by developers using Vertex AI services that require fetching external URLs for analysis or model inputs. Overall activity is low to moderate, aligned with customer workloads rather than large-scale search indexing. RobotSense verifies Google-CloudVertexBot using Google’s official validation methods, ensuring only genuine Google-CloudVertexBot traffic is identified.

Google-Extended is a special user-agent that allows website owners to control whether their publicly accessible content can be used to train and improve Google’s AI models, including products like Gemini. It does not crawl the web itself; instead, it serves as a policy signal interpreted by Google’s AI systems. Site owners can allow or block AI training access by configuring robots.txt rules for Google-Extended. Blocking this agent does not affect Google Search ranking, crawling, or indexing. Its purpose is purely governance-giving publishers a transparent way to manage how their content contributes to Google’s AI research and model development.

Google-InspectionTool is Google’s crawler that simulates how Google Search fetches and renders pages during diagnostics, testing, and site inspection. It is triggered by tools like the Google Search Console URL Inspection tool, Rich Results Test, and Mobile-Friendly Test. The bot fetches pages to evaluate indexing eligibility, structured data, mobile usability, and rendering behavior. Crawl activity is typically on-demand and low volume, generated when site owners or automated systems request diagnostic checks. RobotSense.io verifies Google-InspectionTool using Google’s official validation methods, ensuring only genuine Google-InspectionTool traffic is identified.

Google-Safety is Google’s crawler used to support security, malware detection, and safe-browsing evaluations across the web. It performs targeted checks to detect harmful content, phishing signals, unwanted software, and compromised pages. These scans help maintain Google Safe Browsing warnings and protect users across Chrome, Search, and other Google products. This bot does not respect robots.txt, as that would limit Google’s ability to assess site safety accurately. Crawl activity is typically lightweight and periodic, triggered by risk indicators, user reports, or automated systems monitoring for changes in a site’s security posture. It ignores the global user agent (*) rule. RobotSense.io verifies Google-Safety using Google’s official validation methods, ensuring only genuine Google-Safety traffic is identified.

GoogleOther is a general-purpose crawler used by Google for internal research, large-scale data analysis, and non–Search-related fetching. It is part of Google’s secondary crawling infrastructure, designed to offload tasks that don’t require the full capabilities or strict policies of Googlebot. GoogleOther typically performs broad but lower-priority fetches, such as machine learning dataset generation or internal experiments. Its activity is generally lightweight compared to Googlebot and is separate from indexing operations that directly influence Google Search results. RobotSense.io verifies GoogleOther using Google’s official validation methods, ensuring only genuine GoogleOther traffic is identified.

GoogleOther-Image is a specialized image-focused variant of the GoogleOther crawler, used for internal research, large-scale image analysis, and non–Search-related processing. The bot fetches image files and surrounding metadata but does not directly influence Google Images or Search rankings. Activity is usually lightweight and broad, supporting tasks such as dataset generation, model training, or experimental visual analysis within Google’s internal systems. RobotSense.io verifies GoogleOther-Image using Google’s official validation methods, ensuring only genuine GoogleOther-Image traffic is identified.

GoogleOther-Video is Google’s secondary video-focused crawler, used for internal research, large-scale video analysis, and non-Search-facing processing tasks. The crawler fetches video files, thumbnails, transcripts, and related metadata but does not directly influence Google Search or Google Videos indexing. Crawl activity is typically lightweight, broad, and oriented toward tasks like dataset creation, model evaluation, or experimental video understanding within Google’s internal systems, separate from the responsibilities of Googlebot-Video. RobotSense.io verifies GoogleOther-Video using Google’s official validation methods, ensuring only genuine GoogleOther-Video traffic is identified.

Googlebot is Google’s primary web crawler, responsible for discovering, fetching, and updating content across the public internet for inclusion in Google Search. It operates at massive scale, continuously revisiting sites based on their importance, freshness, and user demand. Googlebot uses a distributed crawling infrastructure that intelligently balances crawl frequency with server load, aiming to gather the most useful and up-to-date information without overwhelming websites. It identifies itself with the Googlebot user-agent family and is fully transparent about its behavior. Genuine Googlebot traffic can be verified through Google’s published reverse-DNS method, which confirms whether an IP truly belongs to Google’s crawling network. Beyond standard HTML pages, Googlebot is capable of rendering JavaScript, interpreting structured data, and evaluating mobile friendliness, which directly influences how pages appear in search results. Googlebot has 2 internal variants i.e., Googlebot Smartphone and Googlebot Desktop. Google increasingly uses Googlebot Smartphone for content crawling. RobotSense.io verifies Googlebot using Google’s official validation methods, ensuring only genuine Googlebot traffic is identified.

Googlebot-Image is Google’s dedicated crawler for discovering and indexing images for Google Images and other visual search features. The bot fetches image files, logos, and favicons and reads surrounding context—alt text, captions, structured data, and page content—to understand relevance and quality. Traffic is usually lightweight and focused on assets rather than full pages. Its role is to keep Google’s image index accurate, fresh, and aligned with user search intent. RobotSense.io verifies Googlebot Image using Google’s official validation methods, ensuring only genuine Googlebot Image traffic is identified.

Googlebot-News is Google’s crawler dedicated to discovering and indexing news content for Google News and Top Stories. The bot focuses on timely, high-quality journalism, scanning article pages, structured data, headlines, timestamps, authorship, and metadata to assess relevance and freshness. Crawling is more frequent than standard Googlebot, reflecting the need for rapid updates. Its role is to ensure accurate, real-time coverage of news sources across Google’s search and news platforms. RobotSense.io verifies Googlebot News using Google’s official validation methods, ensuring only genuine Googlebot News traffic is identified.

Googlebot-Video/1.0 is Google’s specialized crawler for discovering and indexing video content across the web. The bot scans pages for embedded video files, metadata, structured data, thumbnails, transcripts, and video sitemaps to understand relevance, quality, and playback details. Crawl behavior is generally targeted and low-impact, focusing on URLs that contain or reference video assets. Its purpose is to ensure accurate, up-to-date representation of video content in Google Search and related surfaces. RobotSense.io verifies Googlebot Video using Google’s official validation methods, ensuring only genuine Googlebot Video traffic is identified.

[This crawler is officially retired as per Google] AdsBot-Google-Mobile-Apps is Google’s crawler designed to evaluate landing pages used in mobile app promotion campaigns on Google Ads. It simulates a mobile app–driven environment to assess page load speed, relevance, usability, and compliance with Google Ads policies. These evaluations directly influence ad quality scores and campaign performance. Blocking it may prevent Google from validating landing pages for app ads, potentially lowering ad effectiveness or eligibility. Crawl activity is targeted and on-demand, typically occurring when app promotion ads are created, updated, or actively running. It ignores the global user agent (*) rule. RobotSense.io verifies Mobile Apps Android / AdsBot-Google-Mobile-Apps using Google’s official validation methods, ensuring only genuine Mobile Apps Android / AdsBot-Google-Mobile-Apps traffic is identified.

[This crawler is officially retired as per Google] googleweblight is a Google fetcher used by the now-deprecated Google Web Light service, which provided simplified, faster-loading versions of webpages for slow mobile networks. It requested pages to generate lightweight, transcoded versions optimized for low-bandwidth conditions. Site owners allowed it to ensure better accessibility for users on slow connections. Since Web Light has been discontinued, activity from this user-agent is now rare or legacy in nature. Any remaining traffic is typically minimal and related to leftover systems or outdated client requests rather than active Google services. It ignored the global user agent (*) rule. RobotSense.io verifies Web Light / googleweblight using Google’s official validation methods, ensuring only genuine Web Light / googleweblight traffic is identified.