What is Meta-ExternalAgent, and why is it visiting my website?

Meta-ExternalAgent is a crawler operated by Meta that retrieves web content for AI, integrity, safety, and content understanding systems. Unlike Meta's link preview or advertising crawlers, it performs broader content retrieval to help classify, analyze, and understand publicly accessible web pages. Crawl activity is typically initiated by Meta's internal systems rather than user actions and is focused on pages relevant to those systems. For public websites, occasional Meta-ExternalAgent traffic may be expected in server logs.

Is Meta-ExternalAgent a legitimate bot, or is it commonly spoofed?

Meta-ExternalAgent is a legitimate crawler operated by Meta. As with other high-profile crawlers, its User-Agent string can be spoofed by third parties seeking to disguise scraping activity or evade bot detection systems. Attackers may impersonate Meta-operated bots because website administrators sometimes treat recognized crawlers more permissively. User-Agent strings alone cannot verify authenticity and should always be supplemented with network-level verification. You can use Meta's recommended methods mentioned below to verify a legitimate visit, or use RobotSense.io API to easily verify Meta-ExternalAgent bot visits.

How can I verify that a request is really coming from Meta-ExternalAgent?

You can use Meta's recommended official methods to verify Meta-ExternalAgent bot visits, these include: - IP range checks Do not use User-Agent based detection as that can be easily spoofed. Alternatively, you can use RobotSense.io API to easily verify Meta-ExternalAgent bot and all other bots from Meta.

Should I allow or block Meta-ExternalAgent on my website?

Whether to allow Meta-ExternalAgent depends on your content-sharing and data usage preferences. Allowing the crawler enables Meta's systems to access publicly available content for analysis, classification, and related internal uses. Blocking may be appropriate when: - Content is proprietary or licensed. - AI-related content collection is not desired. - Server resources are constrained. - Internal applications or APIs should not be accessed by external crawlers. For many website owners, allowing or blocking Meta-ExternalAgent is primarily a policy decision rather than an SEO decision.

How can I control or block Meta-ExternalAgent using robots.txt or other methods?

You can add a rule in your robots.txt, as given above to control (crawl-delay) or disallow Meta-ExternalAgent bot. The Meta-ExternalAgent bot honors it's own specific robots.txt directives, but does not honor global directives. Also, you can use further controls in your WAF, or in RobotSense enforcement settings to manage the bot behavior.

How often does Meta-ExternalAgent crawl websites, and can it impact server performance?

Meta-ExternalAgent performs ongoing targeted crawling rather than broad search-engine-style indexing. Crawl frequency varies based on Meta's internal content analysis and data collection requirements. Typical impact includes: - Additional bandwidth usage. - Moderate numbers of server requests. - Increased load on dynamic pages when content is retrieved. For most websites, the impact is modest. Larger publishers or frequently updated sites may observe more noticeable bot traffic in website logs.

What happens if I block Meta-ExternalAgent? SEO, visibility, and feature impact explained.

Blocking Meta-ExternalAgent does not affect traditional search engine rankings because Meta does not operate a public web search engine. Potential impacts include: - Reduced access to your content by Meta's AI and content understanding systems. - Limited use of your content in Meta's classification or integrity workflows. - Reduced availability of newly published content to systems that rely on Meta-ExternalAgent crawling. Blocking Meta-ExternalAgent does not directly affect: - Google indexing. - Bing indexing. - Organic search rankings. - Most SEO tools and databases.

Does Meta-ExternalAgent collect, scrape, or use my content for training or reuse?

Yes. Meta-ExternalAgent retrieves publicly accessible webpage content, metadata, and related resources for content understanding, classification, safety, integrity, and AI-related systems. Unlike social preview crawlers, its purpose extends beyond generating link previews or validating ad destinations. Documented uses may include: - Content classification. - Safety and integrity analysis. - Content understanding systems. - AI and machine learning workflows. Meta has publicly described Meta-ExternalAgent as supporting AI-related functions, which may include model training or related data processing. Website owners who do not want their content accessed by this crawler should implement explicit crawler-specific controls rather than relying solely on global robots.txt directives.

Meta-ExternalAgent

Name: Meta-ExternalAgent
Author: Meta / Facebook

AI Training

Operated by Meta / FacebookAI Training

Visit Bot Homepage

Verify Meta-ExternalAgent IP Address

Verify if an IP address truly belongs to Meta / Facebook, using official verification methods. Enter both IP address and User-Agent from your logs for the most accurate bot verification.

Meta-ExternalAgent is a Meta crawler used to fetch webpage content for AI, integrity, and content understanding systems that operate outside classic social preview or ads workflows. It performs broader content retrieval to support tasks like classification, safety analysis, and model training. This traffic is not user-triggered and is separate from Meta’s ad review or link preview bots. Crawl activity is moderate and targeted toward pages relevant to Meta’s internal systems. It does not affect search rankings, as Meta has no public web search engine. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-ExternalAgent using Meta’s official validation methods, ensuring only genuine Meta-ExternalAgent traffic is identified.

This bot does not honor Crawl-Delay rule.

User Agent Examples

Contains: meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Contains: meta-externalagent/1.1

Robots.txt Configuration for Meta-ExternalAgent

Robots.txt User-Agent:Meta-ExternalAgent

Use this identifier in your robots.txt User-agent directive to target Meta-ExternalAgent.

Recommended Configuration

Our recommended robots.txt configuration for Meta-ExternalAgent:

User-agent: Meta-ExternalAgent
Allow: /

Completely Block Meta-ExternalAgent

Prevent this bot from crawling your entire site:

User-agent: Meta-ExternalAgent
Disallow: /

Completely Allow Meta-ExternalAgent

Allow this bot to crawl your entire site:

User-agent: Meta-ExternalAgent
Allow: /

Block Specific Paths

Block this bot from specific directories or pages:

User-agent: Meta-ExternalAgent
Disallow: /private/
Disallow: /admin/
Disallow: /api/

Allow Only Specific Paths

Block everything but allow specific directories:

User-agent: Meta-ExternalAgent
Disallow: /
Allow: /public/
Allow: /blog/

Set Crawl Delay

Limit how frequently Meta-ExternalAgent can request pages (in seconds):

User-agent: Meta-ExternalAgent
Allow: /
Crawl-delay: 10

Note: This bot does not officially mention about honoring Crawl-Delay rule.

Frequently Asked Questions

What is Meta-ExternalAgent, and why is it visiting my website?: Meta-ExternalAgent is a crawler operated by Meta that retrieves web content for AI, integrity, safety, and content understanding systems. Unlike Meta's link preview or advertising crawlers, it performs broader content retrieval to help classify, analyze, and understand publicly accessible web pages. Crawl activity is typically initiated by Meta's internal systems rather than user actions and is focused on pages relevant to those systems. For public websites, occasional Meta-ExternalAgent traffic may be expected in server logs.
Is Meta-ExternalAgent a legitimate bot, or is it commonly spoofed?: Meta-ExternalAgent is a legitimate crawler operated by Meta. As with other high-profile crawlers, its User-Agent string can be spoofed by third parties seeking to disguise scraping activity or evade bot detection systems. Attackers may impersonate Meta-operated bots because website administrators sometimes treat recognized crawlers more permissively. User-Agent strings alone cannot verify authenticity and should always be supplemented with network-level verification. You can use Meta's recommended methods mentioned below to verify a legitimate visit, or use RobotSense.io API to easily verify Meta-ExternalAgent bot visits.
How can I verify that a request is really coming from Meta-ExternalAgent?: You can use Meta's recommended official methods to verify Meta-ExternalAgent bot visits, these include: - IP range checks Do not use User-Agent based detection as that can be easily spoofed. Alternatively, you can use RobotSense.io API to easily verify Meta-ExternalAgent bot and all other bots from Meta.
Should I allow or block Meta-ExternalAgent on my website?: Whether to allow Meta-ExternalAgent depends on your content-sharing and data usage preferences. Allowing the crawler enables Meta's systems to access publicly available content for analysis, classification, and related internal uses. Blocking may be appropriate when: - Content is proprietary or licensed. - AI-related content collection is not desired. - Server resources are constrained. - Internal applications or APIs should not be accessed by external crawlers. For many website owners, allowing or blocking Meta-ExternalAgent is primarily a policy decision rather than an SEO decision.
How can I control or block Meta-ExternalAgent using robots.txt or other methods?: You can add a rule in your robots.txt, as given above to control (crawl-delay) or disallow Meta-ExternalAgent bot. The Meta-ExternalAgent bot honors it's own specific robots.txt directives, but does not honor global directives. Also, you can use further controls in your WAF, or in RobotSense enforcement settings to manage the bot behavior.
How often does Meta-ExternalAgent crawl websites, and can it impact server performance?: Meta-ExternalAgent performs ongoing targeted crawling rather than broad search-engine-style indexing. Crawl frequency varies based on Meta's internal content analysis and data collection requirements. Typical impact includes: - Additional bandwidth usage. - Moderate numbers of server requests. - Increased load on dynamic pages when content is retrieved. For most websites, the impact is modest. Larger publishers or frequently updated sites may observe more noticeable bot traffic in website logs.
What happens if I block Meta-ExternalAgent? SEO, visibility, and feature impact explained.: Blocking Meta-ExternalAgent does not affect traditional search engine rankings because Meta does not operate a public web search engine. Potential impacts include: - Reduced access to your content by Meta's AI and content understanding systems. - Limited use of your content in Meta's classification or integrity workflows. - Reduced availability of newly published content to systems that rely on Meta-ExternalAgent crawling. Blocking Meta-ExternalAgent does not directly affect: - Google indexing. - Bing indexing. - Organic search rankings. - Most SEO tools and databases.
Does Meta-ExternalAgent collect, scrape, or use my content for training or reuse?: Yes. Meta-ExternalAgent retrieves publicly accessible webpage content, metadata, and related resources for content understanding, classification, safety, integrity, and AI-related systems. Unlike social preview crawlers, its purpose extends beyond generating link previews or validating ad destinations. Documented uses may include: - Content classification. - Safety and integrity analysis. - Content understanding systems. - AI and machine learning workflows. Meta has publicly described Meta-ExternalAgent as supporting AI-related functions, which may include model training or related data processing. Website owners who do not want their content accessed by this crawler should implement explicit crawler-specific controls rather than relying solely on global robots.txt directives.