YandexVideo

YaDirectFetcher is a Yandex crawler used to fetch landing pages and assets for advertising purposes within the Yandex.Direct ecosystem. It retrieves content to evaluate ad relevance, page availability, redirects, and compliance with advertising policies. This crawler supports both initial ad review and ongoing campaign monitoring. It is not a general search crawler and does not index content for Yandex Search. Blocking it may disrupt ad approval or delivery. Crawl activity is targeted and low-volume, typically triggered by ad creation, updates, or periodic quality checks across Yandex.Direct campaigns. It does not use the robots.txt at all and thus global robots.txt user agent (*) rule and its own directives are completely ignored. RobotSense.io verifies YaDirectFetcher using Yandex’s official validation methods, ensuring only genuine YaDirectFetcher traffic is identified.

YandexAccessibilityBot is a crawler operated by Yandex to evaluate websites for accessibility and usability compliance. It scans pages to assess factors such as semantic structure, text alternatives, contrast, and navigability for assistive technologies. The insights gathered help Yandex improve accessibility-related features and guidance within its products. This bot does not perform full search indexing and does not directly influence Yandex search rankings. It ignores the global robots.txt user agent (*) rule. Crawl activity is generally lightweight and focused on representative pages rather than comprehensive site-wide crawling. As per Yandex, it sends up to 3 requests to the site per second. The robot ignores the setting in Yandex Webmaster. RobotSense.io verifies YandexAccessibilityBot using Yandex’s official validation methods, ensuring only genuine YandexAccessibilityBot traffic is identified.

YandexAdditional is a Yandex service crawler used to interpret and apply robots.txt rules that control whether indexed page content can be used in Yandex AI-generated responses. It operates only on pages that have already been indexed by Yandex’s primary crawler and does not request new pages or trigger indexing. The bot analyzes access directives to ensure content usage complies with site owner preferences in AI features. It does not perform independent crawling. Its activity is internal and policy-driven, supporting correct enforcement of robots-based restrictions for Yandex’s AI response systems. It ignores the global robots.txt user agent (*) rule. RobotSense.io verifies YandexAdditional using Yandex’s official validation methods, ensuring only genuine YandexAdditional traffic is identified.

YandexAdditionalBot is a Yandex service crawler used to interpret and apply robots.txt rules that control whether indexed page content can be used in Yandex AI-generated responses. It operates only on pages that have already been indexed by Yandex’s primary crawler and does not request new pages or trigger indexing. The bot analyzes access directives to ensure content usage complies with site owner preferences in AI features. Its activity is internal and policy-driven, supporting correct enforcement of robots-based restrictions for Yandex’s AI response systems. It ignores the global robots.txt user agent (*) rule. RobotSense.io verifies YandexAdditionalBot using Yandex’s official validation methods, ensuring only genuine YandexAdditionalBot traffic is identified.

YandexAdNet is Yandex’s advertising-related crawler used to evaluate webpages that participate in the Yandex Advertising Network. It fetches pages to analyze content, layout, ad placement, and policy compliance. These checks help Yandex determine ad relevance, safety, and monetization eligibility for publisher sites. The bot is not a general search crawler and does not index content for Yandex Search. Blocking it may limit ad review or delivery. Crawl activity is targeted and low-volume, typically triggered by publisher onboarding, configuration changes, or ongoing ad quality assessments. RobotSense.io verifies YandexAdNet using Yandex’s official validation methods, ensuring only genuine YandexAdNet traffic is identified.

YandexBlogs is Yandex’s crawler dedicated to discovering and indexing blog-style content for Yandex’s blog and social content aggregations. It focuses on comments in posts, articles, feeds, and frequently updated pages rather than full websites. The bot analyzes text, timestamps, authorship, and update frequency to surface fresh and relevant blog content. Crawl activity is periodic and content-focused, prioritizing sources that publish regularly. Its purpose is to keep Yandex’s blog-oriented services current and reflective of ongoing discussions across the web. RobotSense.io verifies YandexBlogs using Yandex’s official validation methods, ensuring only genuine YandexBlogs traffic is identified.

by Amazon

[Amazon Bots can take upto 30 days to read your Robots.txt updates.] Amzn-SearchBot is Amazon’s web crawler used to discover and retrieve publicly available content for Amazon search and AI-related services. It fetches webpages to analyze text, metadata, and structured information that can support Amazon’s search features and machine learning systems. Crawl activity is typically moderate and focused on publicly accessible pages. Its purpose is to help Amazon improve content discovery, relevance, and information retrieval across its ecosystem. It ignores the global user agent (*) rule. RobotSense.io verifies Amzn-SearchBot using Amazon’s official validation methods, ensuring only genuine Amzn-SearchBot traffic is identified.

by Apple

Applebot is Apple's official web crawler used to power search and content features across Apple services such as Siri, Spotlight Suggestions, and Safari. It crawls webpages to discover content, metadata, and structured information that enhance on-device and cloud-based search experiences. Crawl activity is generally moderate and focused on high-quality, publicly accessible content. Its purpose is to improve search relevance, answers, and suggestions across Apple’s ecosystem without operating a standalone public web search engine. Data crawled by Applebot may be utilized by Apple for foundational model training. Apple allows site owners to opt-out of having their content used for generative model training by disallowing Applebot-Extended in the robots.txt file. RobotSense.io verifies Applebot using Apple's official validation methods, ensuring only genuine Applebot traffic is identified.

by Microsoft

Bingbot is Microsoft’s primary web crawler, responsible for discovering and indexing content for Bing Search and other Microsoft services. The crawler fetches HTML, structured data, images, and metadata to understand page relevance and ranking signals. Crawl activity varies based on site authority, update frequency, and sitemap signals. Its purpose is to keep Bing’s search index fresh, accurate, and aligned with user search intent across Microsoft platforms. RobotSense.io verifies Bingbot using Microsoft’s official validation methods, ensuring only genuine Bingbot traffic is identified.

by Microsoft

BingVideoPreview is Microsoft’s crawler for fetching video-related content to generate previews, thumbnails, and metadata for Bing’s video search experiences. It retrieves video files, poster images, structured data, captions, and surrounding context. This crawler does not perform full-site indexing; instead, it focuses specifically on video assets and the information required to power Bing’s video carousels and preview interfaces. Activity is targeted and relatively low-volume, driven by pages that contain or reference video content. RobotSense.io verifies BingVideoPreview using Microsoft’s official validation methods, ensuring only genuine BingVideoPreview traffic is identified.

by Google

[This crawler is officially retired as per Google] Google Favicon is a specialized Google crawler that retrieves website favicons for use across Google Search, Chrome, and other Google products. It fetches small icon files such as favicon.ico or declared alternative icons in HTML. This bot does not index page content or affect Search rankings; its role is purely to collect icons that visually represent sites in SERPs and browser surfaces. Most sites allow it since its requests are lightweight. Crawl activity is minimal and typically occurs when Google detects new or updated favicon assets. RobotSense.io verifies Google Favicon using Google’s official validation methods, ensuring only genuine Google Favicon traffic is identified.

by Google

Google Publisher Center is a platform that allows news publishers to manage how their content appears across Google News surfaces. When publishers submit feeds, sections, or site updates, Google may fetch associated URLs using Publisher Center–related user-agents to verify content, metadata, and feed accuracy. These fetches are not broad crawls; they are targeted checks tied to publisher actions such as updating feeds, article structures, or publication settings. Blocking it can disrupt feed validation or delay updates in Google News. Activity is typically light, triggered by publisher configuration changes or system refresh cycles. It ignores robots.txt rules. RobotSense.io verifies Google Publisher Center / GoogleProducer using Google’s official validation methods, ensuring only genuine Google Publisher Center / GoogleProducer traffic is identified.