Meta / Facebook

FacebookExternalHit is Facebook’s (Meta’s) crawler used to fetch webpage content for link previews across Facebook, Messenger, Instagram, and other Meta surfaces. It retrieves metadata such as Open Graph tags, titles, descriptions, images, and structured data. These requests are user-triggered, occurring when someone shares or pastes a URL on a Meta platform. The bot does not index or rank websites and has no connection to search algorithms. Blocking it may prevent accurate link previews. Crawl activity is lightweight and focused on fetching just enough content to generate rich social previews. It ignores the global user agent (*) rule. RobotSense.io verifies FacebookExternalHit using Meta’s official validation methods, ensuring only genuine FacebookExternalHit traffic is identified.

Meta-ExternalAds is Meta’s crawler used to evaluate landing pages associated with ads running on Facebook, Instagram, and other Meta platforms. It performs targeted checks to assess page load behavior, policy compliance, redirects, content quality, and overall ad safety. These fetches are ad-driven, not general web crawling, and help Meta determine whether landing pages meet advertising standards. Blocking it may affect ad review accuracy or eligibility. Crawl activity is focused, low-volume, and typically triggered when advertisers submit new ads, update creatives, or undergo automated policy reviews. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalAds using Meta’s official validation methods, ensuring only genuine Meta-ExternalAds traffic is identified.

Meta-ExternalAgent is a Meta crawler used to fetch webpage content for AI, integrity, and content understanding systems that operate outside classic social preview or ads workflows. It performs broader content retrieval to support tasks like classification, safety analysis, and model training. This traffic is not user-triggered and is separate from Meta’s ad review or link preview bots. Crawl activity is moderate and targeted toward pages relevant to Meta’s internal systems. It does not affect search rankings, as Meta has no public web search engine. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalAgent using Meta’s official validation methods, ensuring only genuine Meta-ExternalAgent traffic is identified.

Meta-ExternalFetcher is a Meta crawler that retrieves webpage content to support link previews, metadata extraction, and other external content processing tasks across Facebook, Instagram, and related Meta products. It fetches titles, descriptions, images, and structured data required for rendering shared links or enriching user interactions. These requests are typically user-driven but may also support automated metadata refreshes. Crawl volume is lightweight and focused, targeting only the URLs needed for previews or content enrichment within Meta’s ecosystem. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalFetcher using Meta’s official validation methods, ensuring only genuine Meta-ExternalFetcher traffic is identified.

Meta-WebIndexer is Meta’s web crawler used to discover and fetch publicly available webpage content for internal indexing, AI research, and content understanding tasks. It performs broader, more systematic crawling than Facebook’s preview-focused bots. The crawler analyzes text, metadata, and structured elements to improve Meta’s machine learning models and content classification systems. Crawl activity ranges from moderate to wide-reaching depending on Meta’s data needs. Meta-WebIndexer does not influence external search rankings, as Meta does not operate a web search engine. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-WebIndexer using Meta’s official validation methods, ensuring only genuine Meta-WebIndexer traffic is identified.