M

Meta-WebIndexer

Operated by Meta / FacebookAI Training
Visit Bot Homepage

Verify Meta-WebIndexer IP Address

Verify if an IP address truly belongs to Meta / Facebook, using official verification methods. Enter both IP address and User-Agent from your logs for the most accurate bot verification.

Meta-WebIndexer is Meta’s web crawler used to discover and fetch publicly available webpage content for internal indexing, AI research, and content understanding tasks. It performs broader, more systematic crawling than Facebook’s preview-focused bots. The crawler analyzes text, metadata, and structured elements to improve Meta’s machine learning models and content classification systems. Crawl activity ranges from moderate to wide-reaching depending on Meta’s data needs. Meta-WebIndexer does not influence external search rankings, as Meta does not operate a web search engine. It ignores the global user agent (*) rule. RobotSense.io verifies Meta-WebIndexer using Meta’s official validation methods, ensuring only genuine Meta-WebIndexer traffic is identified.

This bot does not honor Crawl-Delay rule.

User Agent Examples

Contains: meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Contains: meta-webindexer/1.1
Example user agent strings for Meta-WebIndexer

Robots.txt Configuration for Meta-WebIndexer

Robots.txt User-Agent:Meta-WebIndexer

Use this identifier in your robots.txt User-agent directive to target Meta-WebIndexer.

Recommended Configuration

Our recommended robots.txt configuration for Meta-WebIndexer:

User-agent: Meta-WebIndexer
Allow: /

Completely Block Meta-WebIndexer

Prevent this bot from crawling your entire site:

User-agent: Meta-WebIndexer
Disallow: /

Completely Allow Meta-WebIndexer

Allow this bot to crawl your entire site:

User-agent: Meta-WebIndexer
Allow: /

Block Specific Paths

Block this bot from specific directories or pages:

User-agent: Meta-WebIndexer
Disallow: /private/
Disallow: /admin/
Disallow: /api/

Allow Only Specific Paths

Block everything but allow specific directories:

User-agent: Meta-WebIndexer
Disallow: /
Allow: /public/
Allow: /blog/

Set Crawl Delay

Limit how frequently Meta-WebIndexer can request pages (in seconds):

User-agent: Meta-WebIndexer
Allow: /
Crawl-delay: 10

Note: This bot does not officially mention about honoring Crawl-Delay rule.