What is Meta-ExternalAgent, and why is it visiting my website?

Meta-ExternalAgent is a crawler operated by Meta that retrieves web content for AI, integrity, safety, and content understanding systems. Unlike Meta's link preview or advertising crawlers, it performs broader content retrieval to help classify, analyze, and understand publicly accessible web pages. Crawl activity is typically initiated by Meta's internal systems rather than user actions and is focused on pages relevant to those systems. For public websites, occasional Meta-ExternalAgent traffic may be expected in server logs.

Is Meta-ExternalAgent a legitimate bot, or is it commonly spoofed?

Meta-ExternalAgent is a legitimate crawler operated by Meta. As with other high-profile crawlers, its User-Agent string can be spoofed by third parties seeking to disguise scraping activity or evade bot detection systems. Attackers may impersonate Meta-operated bots because website administrators sometimes treat recognized crawlers more permissively. User-Agent strings alone cannot verify authenticity and should always be supplemented with network-level verification. You can use Meta's recommended methods mentioned below to verify a legitimate visit, or use RobotSense.io API to easily verify Meta-ExternalAgent bot visits.

How can I verify that a request is really coming from Meta-ExternalAgent?

You can use Meta's recommended official methods to verify Meta-ExternalAgent bot visits, these include: - IP range checks Do not use User-Agent based detection as that can be easily spoofed. Alternatively, you can use RobotSense.io API to easily verify Meta-ExternalAgent bot and all other bots from Meta.

Should I allow or block Meta-ExternalAgent on my website?

Whether to allow Meta-ExternalAgent depends on your content-sharing and data usage preferences. Allowing the crawler enables Meta's systems to access publicly available content for analysis, classification, and related internal uses. Blocking may be appropriate when: - Content is proprietary or licensed. - AI-related content collection is not desired. - Server resources are constrained. - Internal applications or APIs should not be accessed by external crawlers. For many website owners, allowing or blocking Meta-ExternalAgent is primarily a policy decision rather than an SEO decision.

How can I control or block Meta-ExternalAgent using robots.txt or other methods?

You can add a rule in your robots.txt, as given above to control (crawl-delay) or disallow Meta-ExternalAgent bot. The Meta-ExternalAgent bot honors it's own specific robots.txt directives, but does not honor global directives. Also, you can use further controls in your WAF, or in RobotSense enforcement settings to manage the bot behavior.

How often does Meta-ExternalAgent crawl websites, and can it impact server performance?

Meta-ExternalAgent performs ongoing targeted crawling rather than broad search-engine-style indexing. Crawl frequency varies based on Meta's internal content analysis and data collection requirements. Typical impact includes: - Additional bandwidth usage. - Moderate numbers of server requests. - Increased load on dynamic pages when content is retrieved. For most websites, the impact is modest. Larger publishers or frequently updated sites may observe more noticeable bot traffic in website logs.

What happens if I block Meta-ExternalAgent? SEO, visibility, and feature impact explained.

Blocking Meta-ExternalAgent does not affect traditional search engine rankings because Meta does not operate a public web search engine. Potential impacts include: - Reduced access to your content by Meta's AI and content understanding systems. - Limited use of your content in Meta's classification or integrity workflows. - Reduced availability of newly published content to systems that rely on Meta-ExternalAgent crawling. Blocking Meta-ExternalAgent does not directly affect: - Google indexing. - Bing indexing. - Organic search rankings. - Most SEO tools and databases.

Does Meta-ExternalAgent collect, scrape, or use my content for training or reuse?

Yes. Meta-ExternalAgent retrieves publicly accessible webpage content, metadata, and related resources for content understanding, classification, safety, integrity, and AI-related systems. Unlike social preview crawlers, its purpose extends beyond generating link previews or validating ad destinations. Documented uses may include: - Content classification. - Safety and integrity analysis. - Content understanding systems. - AI and machine learning workflows. Meta has publicly described Meta-ExternalAgent as supporting AI-related functions, which may include model training or related data processing. Website owners who do not want their content accessed by this crawler should implement explicit crawler-specific controls rather than relying solely on global robots.txt directives.

Meta-ExternalAgent

Name: Meta-ExternalAgent
Author: Meta / Facebook

AI Training

Operated by Meta / FacebookAI Training

Visit Bot Homepage

Verify Meta-ExternalAgent IP Address

Verify if an IP address truly belongs to Meta / Facebook, using official verification methods. Enter both IP address and User-Agent from your logs for the most accurate bot verification.

Meta-ExternalAgent is a Meta crawler used to fetch webpage content for AI, integrity, and content understanding systems that operate outside classic social preview or ads workflows. It performs broader content retrieval to support tasks like classification, safety analysis, and model training. This traffic is not user-triggered and is separate from Meta’s ad review or link preview bots. Crawl activity is moderate and targeted toward pages relevant to Meta’s internal systems. It does not affect search rankings, as Meta has no public web search engine. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalAgent using Meta’s official validation methods, ensuring only genuine Meta-ExternalAgent traffic is identified.

This bot does not honor Crawl-Delay rule.

User Agent Examples

Contains: meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Contains: meta-externalagent/1.1

Robots.txt Configuration for Meta-ExternalAgent

Robots.txt User-Agent:Meta-ExternalAgent

Use this identifier in your robots.txt User-agent directive to target Meta-ExternalAgent.

Recommended Configuration

Our recommended robots.txt configuration for Meta-ExternalAgent:

User-agent: Meta-ExternalAgent
Allow: /

Completely Block Meta-ExternalAgent

Prevent this bot from crawling your entire site:

User-agent: Meta-ExternalAgent
Disallow: /

Completely Allow Meta-ExternalAgent

Allow this bot to crawl your entire site:

User-agent: Meta-ExternalAgent
Allow: /

Block Specific Paths

Block this bot from specific directories or pages:

User-agent: Meta-ExternalAgent
Disallow: /private/
Disallow: /admin/
Disallow: /api/

Allow Only Specific Paths

Block everything but allow specific directories:

User-agent: Meta-ExternalAgent
Disallow: /
Allow: /public/
Allow: /blog/

Set Crawl Delay

Limit how frequently Meta-ExternalAgent can request pages (in seconds):

User-agent: Meta-ExternalAgent
Allow: /
Crawl-delay: 10

Note: This bot does not officially mention about honoring Crawl-Delay rule.

Put these rules to work

Test a URL against these rules Build a full robots.txt

Or validate a robots.txt you already have.

Frequently Asked Questions

What is Meta-ExternalAgent, and why is it visiting my website?: Meta-ExternalAgent is a crawler operated by Meta that retrieves web content for AI, integrity, safety, and content understanding systems. Unlike Meta's link preview or advertising crawlers, it performs broader content retrieval to help classify, analyze, and understand publicly accessible web pages. Crawl activity is typically initiated by Meta's internal systems rather than user actions and is focused on pages relevant to those systems. For public websites, occasional Meta-ExternalAgent traffic may be expected in server logs.
Is Meta-ExternalAgent a legitimate bot, or is it commonly spoofed?: Meta-ExternalAgent is a legitimate crawler operated by Meta. As with other high-profile crawlers, its User-Agent string can be spoofed by third parties seeking to disguise scraping activity or evade bot detection systems. Attackers may impersonate Meta-operated bots because website administrators sometimes treat recognized crawlers more permissively. User-Agent strings alone cannot verify authenticity and should always be supplemented with network-level verification. You can use Meta's recommended methods mentioned below to verify a legitimate visit, or use RobotSense.io API to easily verify Meta-ExternalAgent bot visits.
How can I verify that a request is really coming from Meta-ExternalAgent?: You can use Meta's recommended official methods to verify Meta-ExternalAgent bot visits, these include: - IP range checks Do not use User-Agent based detection as that can be easily spoofed. Alternatively, you can use RobotSense.io API to easily verify Meta-ExternalAgent bot and all other bots from Meta.
Should I allow or block Meta-ExternalAgent on my website?: Whether to allow Meta-ExternalAgent depends on your content-sharing and data usage preferences. Allowing the crawler enables Meta's systems to access publicly available content for analysis, classification, and related internal uses. Blocking may be appropriate when: - Content is proprietary or licensed. - AI-related content collection is not desired. - Server resources are constrained. - Internal applications or APIs should not be accessed by external crawlers. For many website owners, allowing or blocking Meta-ExternalAgent is primarily a policy decision rather than an SEO decision.
How can I control or block Meta-ExternalAgent using robots.txt or other methods?: You can add a rule in your robots.txt, as given above to control (crawl-delay) or disallow Meta-ExternalAgent bot. The Meta-ExternalAgent bot honors it's own specific robots.txt directives, but does not honor global directives. Also, you can use further controls in your WAF, or in RobotSense enforcement settings to manage the bot behavior.
How often does Meta-ExternalAgent crawl websites, and can it impact server performance?: Meta-ExternalAgent performs ongoing targeted crawling rather than broad search-engine-style indexing. Crawl frequency varies based on Meta's internal content analysis and data collection requirements. Typical impact includes: - Additional bandwidth usage. - Moderate numbers of server requests. - Increased load on dynamic pages when content is retrieved. For most websites, the impact is modest. Larger publishers or frequently updated sites may observe more noticeable bot traffic in website logs.
What happens if I block Meta-ExternalAgent? SEO, visibility, and feature impact explained.: Blocking Meta-ExternalAgent does not affect traditional search engine rankings because Meta does not operate a public web search engine. Potential impacts include: - Reduced access to your content by Meta's AI and content understanding systems. - Limited use of your content in Meta's classification or integrity workflows. - Reduced availability of newly published content to systems that rely on Meta-ExternalAgent crawling. Blocking Meta-ExternalAgent does not directly affect: - Google indexing. - Bing indexing. - Organic search rankings. - Most SEO tools and databases.
Does Meta-ExternalAgent collect, scrape, or use my content for training or reuse?: Yes. Meta-ExternalAgent retrieves publicly accessible webpage content, metadata, and related resources for content understanding, classification, safety, integrity, and AI-related systems. Unlike social preview crawlers, its purpose extends beyond generating link previews or validating ad destinations. Documented uses may include: - Content classification. - Safety and integrity analysis. - Content understanding systems. - AI and machine learning workflows. Meta has publicly described Meta-ExternalAgent as supporting AI-related functions, which may include model training or related data processing. Website owners who do not want their content accessed by this crawler should implement explicit crawler-specific controls rather than relying solely on global robots.txt directives.

Other Meta / Facebook Bots

Meta / Facebook operates other crawlers you may also need to configure.

FacebookExternalHit

Others

FacebookExternalHit is Facebook’s (Meta’s) crawler used to fetch webpage content for link previews across Facebook, Messenger, Instagram, and other Meta surfaces. It retrieves metadata such as Open Graph tags, titles, descriptions, images, and structured data. These requests are user-triggered, occurring when someone shares or pastes a URL on a Meta platform. The bot does not index or rank websites and has no connection to search algorithms. Blocking it may prevent accurate link previews. Crawl activity is lightweight and focused on fetching just enough content to generate rich social previews. It ignores the global user agent (*) rule. RobotSense.io verifies FacebookExternalHit using Meta’s official validation methods, ensuring only genuine FacebookExternalHit traffic is identified.

Meta-ExternalAds

Ads

Meta-ExternalAds is Meta’s crawler used to evaluate landing pages associated with ads running on Facebook, Instagram, and other Meta platforms. It performs targeted checks to assess page load behavior, policy compliance, redirects, content quality, and overall ad safety. These fetches are ad-driven, not general web crawling, and help Meta determine whether landing pages meet advertising standards. Blocking it may affect ad review accuracy or eligibility. Crawl activity is focused, low-volume, and typically triggered when advertisers submit new ads, update creatives, or undergo automated policy reviews. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalAds using Meta’s official validation methods, ensuring only genuine Meta-ExternalAds traffic is identified.

Meta-ExternalFetcher

Others

Meta-ExternalFetcher is a Meta crawler that retrieves webpage content to support link previews, metadata extraction, and other external content processing tasks across Facebook, Instagram, and related Meta products. It fetches titles, descriptions, images, and structured data required for rendering shared links or enriching user interactions. These requests are typically user-driven but may also support automated metadata refreshes. Crawl volume is lightweight and focused, targeting only the URLs needed for previews or content enrichment within Meta’s ecosystem. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalFetcher using Meta’s official validation methods, ensuring only genuine Meta-ExternalFetcher traffic is identified.

Meta-WebIndexer

AI Training

Meta-WebIndexer is Meta’s web crawler used to discover and fetch publicly available webpage content for internal indexing, AI research, and content understanding tasks. It performs broader, more systematic crawling than Facebook’s preview-focused bots. The crawler analyzes text, metadata, and structured elements to improve Meta’s machine learning models and content classification systems. Crawl activity ranges from moderate to wide-reaching depending on Meta’s data needs. Meta-WebIndexer does not influence external search rankings, as Meta does not operate a web search engine. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-WebIndexer using Meta’s official validation methods, ensuring only genuine Meta-WebIndexer traffic is identified.

Similar Bots

Other AI Training bots from different operators.

Google-Extended

AI Training

by Google

Google-Extended is a special user-agent that allows website owners to control whether their publicly accessible content can be used to train and improve Google’s AI models, including products like Gemini. It does not crawl the web itself; instead, it serves as a policy signal interpreted by Google’s AI systems. Site owners can allow or block AI training access by configuring robots.txt rules for Google-Extended. Blocking this agent does not affect Google Search ranking, crawling, or indexing. Its purpose is purely governance-giving publishers a transparent way to manage how their content contributes to Google’s AI research and model development.

GPTBot

AI Training

by OpenAI

GPTBot is an AI Training bot operated by OpenAI. GPTBot is used by OpenAI to make their generative AI foundation models more useful and safe. It is used to crawl content that may be used in training their generative AI foundation models. Disallowing GPTBot indicates that a site’s content should not be used in training generative AI foundation models. RobotSense.io verifies OpenAI GPTBot using OpenAI’s official validation methods, ensuring only genuine GPTBot traffic is identified.